Manga translation device

Abstract

This manga translation device comprises: an acquisition unit that, on the basis of a manga image, acquires the content of speech, in a first language, included in the manga image and speech-related information related to the speech; a determination unit that, on the basis of the content of the speech and the speech-related information, determines a translation instruction for instructing the generation of a translated text, in a second language, corresponding to the content of the speech; and an output unit that outputs a translated manga image in which the speech in the manga image is replaced by the translated text, of the speech, obtained by inputting the translation instruction into a large-scale language model.

Description

Comic translation apparatus

[0001] The present invention relates to a comic translation apparatus.

[0002] In comic translation, an image in which dialogues are replaced with translated text is generated based on a page image including dialogues. Techniques for automatically performing such processing using machine translation are known. For example, Patent Document 1 describes a technique for replacing text in a comic image with translated text.

[0003] Japanese Unexamined Patent Application Publication No. 2024-20975

[0004] However, conventional techniques have had a problem that there is room for improvement in translation quality.

[0005] For example, in Patent Document 1, the content of dialogues (text data) is input to machine translation. However, in the case of comics, it is difficult to perform high-quality translation based only on the content of dialogues.

[0006] As a specific example, simple machine translation using AI such as LLM (Large Language Model) has difficulty expressing subtle nuances of dialogues.

[0007] The present invention has been made to solve such problems. An object of the present invention is to generate higher-quality translated text for dialogues in comics.

[0008] An example of the comic translation apparatus according to the present invention comprises: an acquisition unit that acquires, based on a comic image, content of a dialogue in a first language included in the comic image and dialogue-related information related to the dialogue; a determination unit that determines a translation instruction commanding generation of translated text in a second language corresponding to the content of the dialogue, based on the content of the dialogue and the dialogue-related information; an output unit that outputs a translated comic image in which the dialogue in the comic image is replaced with the translated text of the dialogue obtained by inputting the translation instruction into a large language model.

[0009] According to the comic translation apparatus of the present invention, higher-quality translated text can be generated for dialogues in comics.

[0010] Other effects are described by the embodiments and modifications shown in this specification and in the drawings.

[0011] An example of the overall configuration including the manga translation device 10 according to Embodiment 1. Example of a manga image. Example of a translated manga image. Example of the configuration of the manga translation device 10. Example of the configuration of the management terminal 20. A flowchart showing an example of the processing performed by the manga translation device 10 in Figure 1. Example of an analysis command. Example of knowledge regarding the size of the panels and the size of the illustrations. Example of knowledge regarding the speaker and the content of the dialogue. Example of knowledge regarding the speaker's facial expression. Example of knowledge regarding the size of the speech bubble. Example of knowledge regarding the shape of the speech bubble. Example of knowledge regarding the type of speaker and the content of the dialogue. Example of knowledge regarding the type of font. Example of knowledge regarding the font size. Example of a translation command.

[0012] Hereinafter, embodiments of the present invention will be described based on the attached drawings.

[0013] [1. Embodiment 1] [1-1. Overall Outline] Figure 1 shows an example of the overall configuration including the manga translation device 10 according to Embodiment 1. The manga translation device 10 is connected to the management terminal 20, VLM 30 (VLM: Visual Language Model or Vision Language Model), and LLM 40 (LLM: Large Language Model) via a communication network.

[0014] The manga translation device 10 acquires manga images from the management terminal 20. The manga images include dialogue. The manga translation device 10 inputs the manga images and analysis commands to the VLM 30, thereby acquiring the content of the dialogue and related information (dialogue-related information) from the VLM 30. The content of the dialogue is text data expressed in the source language (first language), which is natural language. In this embodiment, an example in which the first language is Japanese will be described.

[0015] The manga translation device 10 obtains a translated text of the dialogue from the LLM 40 by inputting a translation command containing the content of the dialogue to the LLM 40. The translated text includes text in the target language (second language) (translated text). In this embodiment, an example in which the second language is English will be described. Hereafter, the term "translated text" will be used as an example of translated text. However, the translated text is not limited to text data corresponding to the term "sentence". For example, the translated text may be a word, a sequence of words, a sentence, etc.

[0016] The manga translation device 10 generates translated manga images by replacing the dialogue in the source language with the translated text of the dialogue in the manga images. The manga translation device 10 outputs the generated translated manga images to the management terminal 20.

[0017] [1-2. Images to be processed] Figure 2 shows a manga image 100 as an example of a manga image. Manga image 100 is an image that represents, for example, at least a part of one page of a manga. Manga image 100 may also be an image that represents two facing pages of a manga. Manga image 100 includes at least one panel 101. Panel 101 may include a speech bubble 102. The speech bubble 102 contains dialogue 103. If there are multiple panels 101, the dialogue 103 is included in one of those panels 101. The dialogue 103 includes at least one of a word, sentence, and text in, for example, a first language. The dialogue 103 is dialogue spoken by speaker 104. In the example in Figure 2, the content of the dialogue 103 is "The Earth is blue".

[0018] Figure 3 shows translated manga image 200 as an example of a translated manga image. The dialogue 103 in the source language has been replaced with the translated dialogue 203 (dialogue in the translated text).

[0019] [1-3. Configuration of the Manga Translation Device] Figure 4 shows an example of the configuration of the manga translation device 10. The manga translation device 10 comprises a processing device 11, a storage device 12, an input device 13, a display device 14, a communication device 15, and a bus 19 that connects these devices to each other.

[0020] The processing unit 11 includes at least one CPU (Central Processing Unit). The at least one CPU is an example of at least one processor. The at least one processor is an example of at least one computer. The manga translation device 10 may also be a computer.

[0021] The storage device 12 is a recording medium that can be read by the processing device 11. The storage device 12 includes at least one memory. The storage device 12 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory) and VRAM (Video Random Access Memory).

[0022] The storage device 12 stores the program PG1. The program PG1 contains at least one instruction. The storage device 12 also stores the knowledge database D1.

[0023] Program PG1 is a program for operating the manga translation device 10. Knowledge database D1 is a database that associates dialogue-related information with the policy for creating translated texts.

[0024] The processing unit 11 reads the program PG1 from the storage device 12. By executing the program PG1, the processing unit 11 functions as an acquisition unit 111, a determination unit 112, and an output unit 113. At least one of the acquisition unit 111, the determination unit 112, and the output unit 113 may be composed of circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0025] The input device 13 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, or sensor). The display device 14 is a display device that displays information to the outside (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel). The input device 13 and the display device 14 may be configured as an integrated unit (e.g., a touch panel).

[0026] The communication device 15 is equipped with a communication interface that can connect to a network and communicates with other devices connected to the network using wireless or wired communication. The communication device 15 may also perform short-range wireless communication such as Bluetooth® or Wi-Fi®.

[0027] Bus 19 is a wiring configuration for communicating information. Bus 19 connects the processing unit 11, the storage device 12, the input device 13, the display device 14, and the communication device 15 to each other. Bus 19 may consist of a single bus, or it may consist of various buses provided between each device.

[0028] [1-4. Configuration of the Management Terminal] Figure 5 shows an example of the configuration of the management terminal 20. The management terminal 20 comprises a processing unit 21, a storage device 22, an input device 23, a display device 24, a communication device 25, and a bus 29 that connects these devices to each other.

[0029] The processing unit 21 includes at least one CPU (Central Processing Unit). The at least one CPU is an example of at least one processor. The at least one processor is an example of at least one computer. The management terminal 20 may also be a computer.

[0030] The storage device 22 is a recording medium that can be read by the processing device 21. The storage device 22 includes at least one memory. The storage device 22 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory) and VRAM (Video Random Access Memory).

[0031] The storage device 22 stores the program PG2. Program PG2 contains at least one instruction. Program PG2 is a program for operating the management terminal 20.

[0032] The processing unit 21 reads the program PG2 from the storage device 22. The processing unit 21 implements the functions described herein by executing the program PG2. At least a portion of the processing unit 21 may be composed of circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0033] The input device 23 is an input device that accepts input from the outside (for example, a keyboard, mouse, microphone, switch, button, or sensor). The display device 24 is a display device that displays information to the outside (for example, various display panels such as a liquid crystal display panel or an organic EL display panel). The input device 23 and the display device 24 may be configured as an integrated unit (for example, a touch panel).

[0034] The communication device 25 is equipped with a communication interface that can connect to a network and communicates with other devices connected to the network using wireless or wired communication. The communication device 25 may also perform short-range wireless communication such as Bluetooth® or Wi-Fi®.

[0035] Bus 29 is a wiring configuration for communicating information. Bus 29 connects the processing unit 21, the storage device 22, the input device 23, the display device 24, and the communication device 25. Bus 29 may consist of a single bus, or it may consist of various buses provided between each device.

[0036] [1-5. VLM Configuration]

[0037] VLM30 receives image data and prompts as input and outputs information about the image content in response to the prompts. VLM30 is a pre-trained model that has been trained, for example, using a large amount of image and text data to output appropriate natural language responses to image and natural language inputs. Various known VLMs can be used as VLM30. For example, DALL-E, CLIP, LLaVA, etc. are known, but are not limited to these.

[0038] In this embodiment, input and output to the VLM30 are performed using the same language as the dialogue 103. That is, in this embodiment, Japanese is used. However, a language different from the language of the dialogue 103 may be used. For example, the target language may be used. Furthermore, other languages ​​may also be used.

[0039] [1-6. LLM Configuration] LLM40 receives natural language data, including prompts, as input and outputs a response in natural language. LLM40 is a pre-trained model that has been trained, for example, using a large amount of text data, to output an appropriate response in natural language to a natural language input. Various publicly known models can be used as LLM40. For example, GPT, BERT, PaLM, Llama, etc. are publicly known, but are not limited to these.

[0040] In this embodiment, input to and output from the LLM 40 is performed using the same language as the language of the dialogue 103. That is, Japanese is used in this embodiment. However, a language different from the language of the dialogue 103 may be used. For example, a target translation language may be used. Furthermore, other languages may be used.

[0041] [1-7. Processing] FIG. 6 is a flowchart illustrating an example of processing executed by the comic translation device 10 according to the present embodiment. This processing is started in response to the acquisition unit 111 of the comic translation device 10 acquiring a comic image 100 (step S1).

[0042] After step S1, the acquisition unit 111 determines an analysis command for performing analysis on the dialogue 103 and the like included in the comic image 100 (step S2). The analysis command includes, for example, a sentence representing the command. Further, the analysis command may include, for example, a sentence representing the command. The analysis command is, for example, text for instructing the VLM 30 to analyze the comic image 100.

[0043] FIG. 7 shows an example of the analysis command. The analysis command includes a sentence for instructing analysis of the comic image 100. In the example of FIG. 7, the sentence for instructing analysis of the comic image 100 includes "Tell me about the following for each panel one by one." Here, a panel 101 may include a dialogue 103.

[0044] The analysis command includes a sentence for describing one or more items to be analyzed. In the example of FIG. 7, four items are described. The items may be hierarchical. In the example of FIG. 7, item 1 consists of only a single hierarchy, item 2 consists of two hierarchies, item 3 consists of two hierarchies, and item 4 consists of two hierarchies.

[0045] Item 1 instructs analysis of the size of a panel 101 that includes the dialogue 103. The size of a panel is expressed, for example, by the proportion of the panel that occupies a page.

[0046] Item 2 commands analysis of the illustration drawn in frame 101. In the example of FIG. 7, analysis is commanded for the name of the character that is the speaker 104 drawn in frame 101 (i.e., the speaker of line 103), the facial expression of the speaker 104, i.e., the emotion, and the objects drawn in frame 101 other than the speaker 104.

[0047] The analysis command may include an example showing the text of the analysis result. In item 2-2 of FIG. 7, after the main text "Please tell us the facial expression of the character who is the speaker.", an example "(Example: joy, anger, sorrow, pleasure, impatience)" is added in parentheses.

[0048] Further, the analysis command may include an instruction to limit the length of the analysis result. In item 2-3 of FIG. 7, after the main text "Please tell us the objects drawn other than the character who is the speaker.", a restriction "(within 30 characters)" is added in parentheses.

[0049] Item 3 commands analysis of speech balloon 102. In the example of FIG. 7, analysis is commanded for the size of speech balloon 102 and the shape of speech balloon 102.

[0050] Further, the analysis command may include an instruction that specifies the expression format of the analysis result. In item 3-1 of FIG. 7, after the main text "Please tell us the size of the speech balloon.", an expression format "(percentage of the frame)" is added in parentheses.

[0051] Item 4 commands analysis of line 103. In the example of FIG. 7, analysis is commanded for the line itself (i.e., text data representing the content of the line), the type of the content of the line, the type of the font of the line, and the size of the font.

[0052] The acquisition unit 111 may dynamically generate an analysis command in accordance with the comic image 100. Alternatively, the acquisition unit 111 may generate the analysis command by simply acquiring a pre-specified analysis command. It is preferable that the content of the analysis command is appropriately designed in advance according to the performance, specifications, etc. of the VLM 30.

[0053] Let's return to the explanation in Figure 6. After step S2, the acquisition unit 111 outputs the cartoon image 100 and the analysis command to the VLM 30 (step S3).

[0054] Here, although not shown in Figure 6, the VLM 30 receives the manga image 100 and an analysis command. The VLM 30 then analyzes the manga image 100 according to the analysis command and outputs the analysis results to the manga translation device 10. The analysis results include the content of the dialogue and information related to the dialogue (dialogue-related information).

[0055] In response, the acquisition unit 111 acquires the content of the dialogue and dialogue-related information from the VLM 30 (step S4). The content of the dialogue is text data representing the content of the dialogue 103. For example, the content of the dialogue is language data in the source language. In this way, the acquisition unit 111 acquires the content of the dialogue and dialogue-related information using the VLM 30.

[0056] In the example in Figure 7, according to item 1, the dialogue-related information includes information representing the size of frame 101.

[0057] According to item 2, the dialogue-related information includes information about the illustration in panel 101. In particular, the dialogue-related information includes information representing the name of speaker 104 (item 2-1), information representing the facial expression of speaker 104 (item 2-2), and information representing the objects depicted in panel 101 (item 2-3). In the modified example, the dialogue-related information does not need to include all of these. For example, the dialogue-related information may include at least one of items 2-1, 2-2, and 2-3.

[0058] According to item 3, the dialogue-related information includes information about the speech bubble 102 related to the dialogue 103. In particular, the dialogue-related information includes information representing the size of the speech bubble 102 and information representing the shape of the speech bubble 102. In the modified example, the dialogue-related information does not need to include all of these. For example, the dialogue-related information includes at least one of items 3-1 and 3-2.

[0059] According to item 4, the dialogue-related information includes information representing the type of content of dialogue 103. The dialogue-related information also includes information regarding the font of dialogue 103. The font information includes information representing the font type and information representing the font size. Note that in the modified version, the dialogue-related information does not need to include all of these. For example, the dialogue-related information includes at least one of items 4-2, 4-3, and 4-4. Note that in the modified version, the dialogue-related information does not need to include all of these. For example, the dialogue-related information includes at least one of items 4-2, 4-3, and 4-4. Note that item 4-1 is an instruction for obtaining the content of the dialogue (i.e., text data).

[0060] In the modified example, the dialogue-related information does not need to include all of items 1 to 4. For example, the dialogue-related information includes at least one of items 1 to 4. However, the item corresponding to item 4-1, i.e., the content of the dialogue (i.e., text data), is commanded to be retrieved as the content of the dialogue even if it is not included in the dialogue-related information.

[0061] Next, the decision unit 112 determines information representing the translation policy (translation policy information) based on the dialogue-related information (step S5). For example, the translation policy information is determined by searching the knowledge database D1 based on the dialogue-related information. The knowledge database D1 associates conditions related to the dialogue-related information with the translation policy information.

[0062] In this way, by determining the creation policy information based on the dialogue-related information, an appropriate creation policy is determined according to the context of the dialogue.

[0063] Figures 8 to 15 show specific examples of the knowledge database D1. Figure 8 is an example of knowledge regarding panel size and illustration size. In the example in Figure 8, the conditions are expressed by combining panel size and illustration size. In this example, the creation policy information includes information indicating the length of the translation to be generated. For example, if the panel size is two pages (i.e., the entire spread) and the illustration is large, it is specified that the translation should be made shorter.

[0064] By using this knowledge, it is possible to generate appropriate translations based on the size of the panels. For example, the length of the translations will be more appropriate. Furthermore, by using this knowledge, it is possible to generate appropriate translations based on the size of the illustrations. For example, the length of the translations will be more appropriate.

[0065] Here, the condition may not include the size of the illustration. Alternatively, if the condition includes the size of the illustration, for example, item 2 of the analysis instruction may instruct the system to obtain the size of the illustration.

[0066] Figure 9 shows an example of knowledge regarding the speaker and the content of the lines. For example, if the speaker is Docomo Hanako and the line is "I'm the strongest!", it is specified that it should be translated as a victory catchphrase.

[0067] In this example, it is specified as a creation policy information that the phrase should be translated as a victory catchphrase. In the modified example, the fact that it is a victory catchphrase may be specified as dialogue-related information. That is, the decision unit 112 may determine new dialogue-related information based on the dialogue-related information.

[0068] By using this knowledge, it is possible to generate appropriate translations based on the speaker's name. For example, the wording of the translation will be more appropriate. Furthermore, by using this knowledge, individual policies can be specified for frequently occurring lines of dialogue, resulting in the generation of even more appropriate translations.

[0069] Figure 10 shows an example of knowledge regarding the speaker's facial expression. For example, if the speaker's facial expression is "joyful," the translation is specified to have a cheerful tone.

[0070] By using this kind of knowledge, it's possible to generate appropriate translations based on the speaker's facial expressions. For example, the speaker's emotions can be expressed more accurately.

[0071] Figure 11 shows an example of knowledge regarding speech bubble size. In this example, the creation policy information includes information indicating the length of the translated text to be generated. For example, if the speech bubble occupies 50% or more of the panel's area, it is specified that the translated text should be longer.

[0072] By using this knowledge, it is possible to generate appropriate translations based on the size of the speech bubble. For example, since larger speech bubbles require longer translations, the length of the translation will be more appropriate.

[0073] Figure 12 shows an example of knowledge regarding speech bubble shapes. For example, if the speech bubble is in the shape of an explosion, the translation is specified to be emotional.

[0074] By using this knowledge, it is possible to generate appropriate translations based on the shape of the speech bubble. For example, the speaker's emotions can be expressed more accurately.

[0075] Figure 13 is an example of knowledge regarding the speaker and the type of dialogue content. In the example in Figure 13, the conditions are expressed by combining the type of dialogue content and the speaker. The type of dialogue content can be expressed, for example, by text data. The type of dialogue content can be appropriately defined by a person skilled in the art depending on the content of the manga to be translated.

[0076] In the example in Figure 13, the creation policy information includes information representing the number of translations to be generated. Here, "number of translations" refers to, for example, the number of candidate translations. Alternatively, "number of translations" refers to the number of proposed translations. Alternatively, "number of translations" refers to the number of different translations corresponding to the same line of dialogue. Alternatively, "number of translations" refers to the number of different translations corresponding to a single sentence contained in the dialogue. Alternatively, "number of translations" refers to the number of translation patterns.

[0077] For example, if the content of the dialogue is a description of an ability and the speaker is Docomo Hanako, it is specified that the number of translations that VLM30 should generate should be 3. In this case, VLM30 will output 3 different translation drafts. Note that in the example in Figure 13, the conditions may not include the speaker.

[0078] The determination unit 112 may change the temperature parameter of the LLM 40 according to the number of translations to be generated. For example, if the number of translations to be generated is larger, the temperature value may be changed to a larger value.

[0079] By using this knowledge, it is possible to generate appropriate translations based on the speaker's name. For example, the wording of the translation will be more appropriate. Furthermore, by using this knowledge, it is possible to generate appropriate translations based on the type of content of the dialogue. For example, for types that are expected to be complex, or types that are expected to express a large amount of information in a short sentence, multiple translation options will be presented, increasing the likelihood that the most appropriate translation will be selected. On the other hand, for types that are expected to be simple, a single translation can be generated, making the processing more efficient.

[0080] Figure 14 shows an example of knowledge regarding font types. For example, if the font is a Gothic font, it is specified that the translated text should be written in a bold style.

[0081] By using this knowledge, it is possible to generate appropriate translations based on the font type. For example, for dialogue written in a font that indicates emphasis (such as a Gothic font), an emphasized translation will be generated.

[0082] Figure 15 shows an example of knowledge regarding font size. For example, if the font size is 10 points or larger, it is specified that the translated text should be made more forceful.

[0083] By using this knowledge, it is possible to generate appropriate translations based on font size. For example, for dialogue written in a large font, an emphasized translation will be generated.

[0084] Figures 8 to 15 are illustrative examples, and it is possible to use knowledge other than that shown. For example, knowledge about information representing the objects depicted in the panels may be used. With such knowledge, the objects mentioned in the dialogue are taken into consideration, which allows for the generation of more appropriate translations.

[0085] In the examples shown in Figures 8 to 15, the production policy information is acquired as information separate from the dialogue-related information. As a variation, the production policy information may include some or all of the dialogue-related information.

[0086] The explanation returns to Figure 6. After step S5, the decision unit 112 determines a translation command based on the creation policy information (step S6). The translation command includes a sentence that instructs the LLM 40 to create a translation corresponding to the content of the dialogue. The translation command may also include a sentence that instructs the LLM 40 to create a translation corresponding to the content of the dialogue. The decision unit 112 determines the translation command based on the content of the dialogue and the dialogue-related information.

[0087] Figure 16 shows an example of a translation command. The translation command includes the content of the dialogue. In the example in Figure 16, the content of the dialogue is "The Earth is blue," which corresponds to dialogue 103 in Figure 2. The translation command may also include production policy information. In the example in Figure 16, it is specified that the dialogue should be short and that five translation drafts should be output. By including production policy information in the translation command, it is possible to appropriately instruct the LLM40 on the policy.

[0088] Translation instructions may include some or all of the dialogue-related information. In the example in Figure 16, the instructions include information indicating the type of dialogue, the speaker's name, and the speaker's facial expression. Translation instructions may also include information indicating the target language. In the example in Figure 16, the target language is specified as English.

[0089] Next, the decision unit 112 outputs a translation command to the LLM 40 (step S7). Here, although not shown in Figure 6, the LLM 40 receives the translation command. The LLM 40 then generates one or more translated lines of dialogue according to the translation command and outputs the one or more generated translated lines to the manga translation device 10. The translated lines are, for example, language data in the target language.

[0090] Accordingly, the determination unit 112 obtains one or more translated sentences from the LLM 40 (step S8).

[0091] Next, the output unit 113 generates a translated manga image 200 by replacing the dialogue 103 in the source language with the translated dialogue 203 in the manga image 100 (step S9). Here, the translated manga image 200 is an image in which the dialogue 103 in the manga image 100 has been replaced with the translated text of the dialogue 103. In the example in Figure 3, the translated dialogue 203 is "The earth is blue." The translated text of the dialogue 103 is the translated text obtained from the LLM 40 by inputting a translation command to the LLM 40, as described above.

[0092] Next, the output unit 113 outputs the translated manga image 200 (step S10). The output is made, for example, to the management terminal 20. Although not shown in Figure 16, after step S10, the management terminal 20 receives the translated manga image 200. After that, the management terminal 20 outputs the translated manga image 200. The output of the translated manga image 200 is done, for example, by display on the display device 24. The management terminal 20 may also store the translated manga image 200 in the storage device 22.

[0093] If multiple translations are obtained in step S8, steps S9 and S10 may be performed for each translation.

[0094] Alternatively, if multiple translations are obtained in step S8, a process for selecting one translation from among the multiple translations may be executed. For example, the output unit 113 may automatically select one translation. A process for automatically selecting one translation can be appropriately designed by a person skilled in the art. For example, other LLMs may be used.

[0095] Alternatively, the output unit 113 may present multiple translations to the user as a process for selecting one translation from among several translations. Furthermore, the output unit 113 may select one translation in response to user input. With this configuration, the user can select one appropriate translation from among the multiple translations presented by the output unit 113.

[0096] Furthermore, the output unit 113 may present the user with multiple translated manga images 200 as a process for selecting one translated text from among several translated texts. In addition, the output unit 113 may select one translated manga image 200 in response to input from the user. With this configuration, the user can select one appropriate translated text from among the multiple translated texts presented by the output unit 113.

[0097] [1-8. Examples of Effects] As described above, the manga translation device 10 according to Embodiment 1 determines translation commands by considering not only the content of the dialogue but also various information related to the dialogue. For this reason, for example, it can reproduce the subtle nuances of the dialogue. As a result, higher quality translations of dialogue in manga are generated.

[0098] [2. Modified Versions] In the above-described embodiment 1, the manga translation device 10 stores the knowledge database D1. In the modified version, the knowledge database D1 may be stored on an external computer. In that case, the manga translation device 10 may transmit dialogue-related information to the external computer. The manga translation device 10 may also receive creation policy information from the external computer.

[0099] In the above-described embodiment 1, the manga translation device 10 does not include VLM30 and LLM40. In a modified example, the manga translation device 10 may include at least one of VLM30 and LLM40.

[0100] [3. Others] (1) Each function illustrated in Figures 4 and 5 can be implemented by any combination of hardware and software. The method of implementing each function is not particularly limited. Each function may be implemented using one device that is physically or logically coupled, or it may be implemented using a device that is configured by directly or indirectly connecting two or more physically or logically separated devices (for example, using wired, wireless, etc.). Each function may be implemented by combining the above one device or the above multiple devices with software.

[0101] (2) In this disclosure, the term “apparatus” may be replaced with other terms such as circuit, device or unit.

[0102] (3) In each embodiment and each modification, the storage device may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., compact disc, digital multipurpose disc, Blu-ray® disc), a smart card, flash memory (e.g., card, stick, key drive), a floppy® disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.

[0103] (4) Each embodiment and each modification is based on LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or decimal), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20 may apply to at least one system utilizing UWB (Ultra-WideBand), Bluetooth®, or other appropriate systems, and to next-generation systems extended, modified, created, or defined based thereon. Alternatively, multiple systems may be applied in combination (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0104] (5) The processing procedures, sequences, or flowcharts illustrated in each embodiment and each modification may be in any order, as long as they do not contradict each other. For example, the methods described in this disclosure present various step elements in an exemplary order and are not limited to the specific order presented.

[0105] (6) In each embodiment and each modification, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0106] (7) In each embodiment and each modification, the determination may be based on a value represented by one bit (0 or 1), on a Boolean value (true or false), or on a numerical comparison (for example, a comparison with a predetermined value).

[0107] (8) The programs illustrated in each embodiment and each modification should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, whether they are called software, firmware, middleware, microcode, or hardware description languages ​​or by other names. Furthermore, software, or instructions, etc., may be transmitted or received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0108] (9) The information described in each embodiment and each modification may be represented using any of the various different technologies. For example, the data, information, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Notwithstanding the terms described herein and the terms necessary for understanding this disclosure, terms may be replaced with terms having the same or similar meanings.

[0109] (10) In each embodiment and each variation, the terms “system” and “network” are used interchangeably.

[0110] (11) In each embodiment and each variation, the terminal is, for example, a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.

[0111] (12) A mobile station may also be called a transmitting device, receiving device, or communication device. A mobile station may also be a device mounted on a mobile body, or the mobile body itself. A mobile body means a movable object. The speed of movement of a mobile body is arbitrary. A mobile body can be stopped. A mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on them. A mobile body may be a mobile body that moves autonomously based on operational commands. A mobile body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station could be an IoT (Internet of Things) device such as a sensor.

[0112] (13) In each embodiment and each variation, the term “decision” may encompass a wide variety of actions. “Decision” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or another data structure), and ascertaining. “Decision” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). “Decision” may also include resolving, selecting, choosing, establishing, and comparing. In other words, "decision" can include considering that some action has been "decided." Furthermore, "decision" can be reinterpreted as "assuming," "expecting," or "considering."

[0113] (14) In each embodiment and each modification, the term “connected,” or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in the present disclosure, two elements may be considered to be “connected” or “coupled” to each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0114] (15) In each embodiment and each modification, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”

[0115] (16) Any reference to elements using the designations “first” and “second” as used in this disclosure does not limit the quantity or order of those elements in general. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be adopted or that the first element must precede the second element in any way.

[0116] (17) Where “include,” “including,” and variations thereof are used in this disclosure or claims in each embodiment and each variation, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, where the term “or” is used in this disclosure or claims, it is intended not to be an exclusive OR.

[0117] (18) Where articles are added in translation, for example, in English, such as a, an, and the, the disclosure may include the fact that the noun following these articles is plural.

[0118] (19) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0119] (20) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combined” may be interpreted in the same way as “different.”

[0120] (21) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of certain information (e.g., notification that "it is X") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0121] (22) It will be obvious to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Accordingly, the descriptions herein are for illustrative purposes only and are not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined.

[0122] [4. Aspects understood from the above embodiments or modifications] The following aspects can be understood from at least one of the above embodiments or modifications.

[0123] [4-1. First Embodiment] The manga translation device according to the first embodiment comprises: an acquisition unit that acquires the content of dialogue in a first language and dialogue-related information related to the dialogue contained in a manga image based on the manga image; a determination unit that determines a translation command that commands the generation of translated text in a second language corresponding to the content of the dialogue, based on the content of the dialogue and the dialogue-related information; and an output unit that outputs a translated manga image in which the dialogue in the manga image is replaced with translated text of the dialogue obtained by inputting the translation command into a large-scale language model.

[0124] According to this method, higher-quality translations of dialogue in manga are generated. For example, compared to simple machine translation using AI such as LLM (Large-Scale Language Model), the subtle nuances of the dialogue are expressed more appropriately.

[0125] [4-2. Second Embodiment] According to the manga translation device of the second embodiment, the manga image includes at least one frame, the dialogue is included in one of the at least one frame, and the dialogue-related information includes information representing the size of the frame containing the dialogue.

[0126] According to this embodiment, an appropriate translation is generated based on the size of the frame. For example, the length of the translation becomes more appropriate.

[0127] [4-3. Third Embodiment] According to the manga translation device of the third embodiment, the manga image includes at least one panel, the dialogue is included in one of the at least one panel, and the dialogue-related information includes at least one of the following: information representing the name of the speaker of the dialogue, information representing the facial expression of the speaker of the dialogue, and information representing an object depicted in the panel containing the dialogue.

[0128] In this embodiment, an appropriate translation is generated based on the speaker's name. For example, the wording of the translation becomes more appropriate. Also, an appropriate translation is generated based on the speaker's facial expression. For example, the speaker's emotions are expressed more appropriately. Also, an appropriate translation is generated based on the object depicted in the panel. For example, the object being referred to in the dialogue is taken into consideration, resulting in a more appropriate translation.

[0129] [4-4. Fourth Embodiment] According to the manga translation device of the fourth embodiment, the dialogue-related information includes at least one of the following: information representing the size of the speech bubble associated with the dialogue, and information representing the shape of the speech bubble associated with the dialogue.

[0130] According to this embodiment, an appropriate translation is generated based on the size of the speech bubble. For example, the length of the translation becomes more appropriate. Also, an appropriate translation is generated based on the shape of the speech bubble. For example, the speaker's emotions are expressed more appropriately.

[0131] [4-5. Fifth Embodiment] According to the manga translation device of the fifth embodiment, the dialogue-related information includes at least one of the following: information representing the type of content of the dialogue, information representing the type of font of the dialogue, and information representing the size of the font of the dialogue.

[0132] In this embodiment, an appropriate translation is generated based on the type of content of the dialogue. For example, for types of dialogue that are expected to be complex, or types of dialogue that are expected to express a large amount of information in a short sentence, multiple translation options are presented, increasing the likelihood of finding a more appropriate translation. Furthermore, an appropriate translation can be generated based on the type of font. For example, for dialogue written in a font that indicates emphasis (such as a Gothic font), an emphasized translation is generated. Additionally, an appropriate translation is generated based on the font size. For example, for dialogue written in a large font, an emphasized translation is generated.

[0133] [4-6. Sixth Embodiment] According to the manga translation device of the sixth embodiment, the dialogue-related information further includes: information about the illustration of the panel containing the dialogue; information about the speech bubble related to the dialogue; information representing the type of content of the dialogue; and information about the font of the dialogue.

[0134] According to this method, an appropriate translation is generated based on various pieces of information.

[0135] [4-7. Seventh Embodiment] According to the manga translation device of the seventh embodiment, the determination unit determines creation policy information representing the creation policy for the translated text based on the dialogue-related information.

[0136] In this configuration, an appropriate writing policy is determined according to the context of the dialogue.

[0137] [4-8. Eighth Embodiment] According to the manga translation device of the eighth embodiment, the translation command includes the creation policy information.

[0138] According to this embodiment, the LLM can be appropriately instructed on the policy.

[0139] [4-9. Ninth Embodiment] According to the manga translation device of the ninth embodiment, the creation policy information includes information representing the number of candidate translation texts.

[0140] In this configuration, the number of translated sentences changes depending on the dialogue-related information, etc., so multiple translation options are presented as needed.

[0141] [4-10. Tenth Embodiment] According to the manga translation device of the tenth embodiment, the creation policy information includes information representing the length of the translation text to be generated.

[0142] According to this embodiment, the length of the translated text changes depending on the dialogue-related information, so that a translated text of an appropriate length is generated.

[0143] 10...Manga translation device, 20...Management terminal, 30...VLM, 40...LLM, 100...Manga image, 101...Panel, 102...Speech bubble, 103...Dialogue, 104...Speaker, 111...Acquisition unit, 112...Decision unit, 113...Output unit, 200...Translated manga image, 203...Translated dialogue, D1...Knowledge database, PG1...Program, PG2...Program.

Claims

1. A manga translation device comprising: an acquisition unit that acquires the content of dialogue in a first language and dialogue-related information related to the dialogue contained in a manga image, based on the manga image; a determination unit that determines a translation command to generate a translated text in a second language corresponding to the content of the dialogue, based on the content of the dialogue and the dialogue-related information; and an output unit that outputs a translated manga image in which the dialogue in the manga image has been replaced with the translated text of the dialogue obtained by inputting the translation command into a large-scale language model.

2. The manga translation device according to claim 1, wherein the manga image includes at least one panel, the dialogue is included in one of the at least one panel, and the dialogue-related information includes information representing the size of the panel containing the dialogue.

3. The manga translation device according to claim 1, wherein the manga image includes at least one panel, the dialogue is included in one of the at least one panel, and the dialogue-related information includes at least one of the following: information representing the name of the speaker of the dialogue, information representing the facial expression of the speaker of the dialogue, and information representing an object depicted in the panel containing the dialogue.

4. The manga translation device according to claim 1, wherein the dialogue-related information includes at least one of the following: information representing the size of the speech bubble associated with the dialogue, and information representing the shape of the speech bubble associated with the dialogue.

5. The manga translation device according to claim 1, wherein the dialogue-related information includes at least one of the following: information representing the type of content of the dialogue, information representing the type of font of the dialogue, and information representing the size of the font of the dialogue.

6. The manga translation device according to claim 2, wherein the dialogue-related information further includes: information about the illustration of the panel containing the dialogue; information about a speech bubble related to the dialogue; information representing the type of content of the dialogue; and information about the font of the dialogue.

7. The manga translation apparatus according to claim 1, wherein the determination unit determines creation policy information representing a policy for creating the translated text based on the dialogue-related information.

8. The manga translation device according to claim 7, wherein the translation instruction includes the creation policy information.

9. The manga translation device according to claim 7, wherein the creation policy information includes information representing the number of candidates for the translated text.

10. The manga translation device according to claim 7, wherein the creation policy information includes information representing the length of the translation text to be generated.

Images