Moving image generation device

Abstract

This moving image generation device comprises: a determination unit that determines, on the basis of first moving image data, a plurality of speech data items corresponding one-to-one to a plurality of speeches included in the first moving image data, each of the plurality of speech data items indicating at least a corresponding speech by text; an information acquisition unit that acquires, from a language model, designation information designating one or more speech data items corresponding to a highlight scene of the first moving image data from among the plurality of speech data items by inputting a prompt including the plurality of speech data items to the language model; and an extraction unit that extracts, on the basis of the designation information and from the first moving image data, second moving image data indicating the highlight scene of the first moving image data.

Description

Video generation device

[0001] The present invention relates to a video generation device.

[0002] In recent years, video distribution services have rapidly spread, and many people have come to watch various contents. In such video distribution services, a technique of extracting a part of video data and distributing the part of video data may be used.

[0003] For example, the video service providing method disclosed in Patent Document 1 separates a video into a plurality of unit intervals and automatically generates keywords for each unit interval. Further, the video service providing method searches for a unit interval corresponding to a keyword input from a user and provides the searched unit interval to the user.

[0004] Japanese Patent No. 6824332

[0005] However, when extracting the highlight scene of the video data as a part of the video data, it is often the case that a person in charge extracts it manually. Selecting the highlight scene manually is time-consuming and cannot quickly respond to a large amount of video processing.

[0006] An object of the present disclosure is to provide a video generation device that can handle a large amount of video processing with less labor when extracting the highlight scene of video data as compared with the prior art.

[0007] The video generation device according to the present disclosure determines a plurality of caption data corresponding one-to-one to a plurality of captions included in the first video data based on the first video data, and each of the plurality of caption data indicates at least the corresponding caption by text; a determination unit; an information acquisition unit that inputs a prompt including the plurality of caption data to a language model, and acquires designation information for designating one or more caption data corresponding to the highlight scene of the first video data from among the plurality of caption data from the language model; and an extraction unit that extracts second video data indicating the highlight scene of the first video data from the first video data based on the designation information.

[0008] According to this disclosure, when extracting highlight scenes from video data, it is possible to handle large amounts of video processing with less effort compared to conventional technologies.

[0009] A block diagram showing an example of the overall configuration of video generation system 1. A block diagram showing an example of the configuration of video generation device 10. A diagram showing an example of text data group TG. A diagram showing an example of prompt PP format. A diagram showing an example of prompt PP. A diagram showing an example of specified information MI. A diagram showing an example of the flow of various data and information used in video generation device 10. A flowchart showing an example of the operation of video generation device 10. A block diagram showing an example of the overall configuration of video generation system 1A according to modification 1.

[0010] 1: The video generation system 1 according to the first embodiment will be described below with reference to Figures 1 to 8.

[0011] 1-1: Configuration of the First Embodiment 1-1-1: Overall Configuration Diagram 1 is a block diagram showing an example of the overall configuration of the video generation system 1 according to this embodiment. As shown in Figure 1, the video generation system 1 comprises a video generation device 10, an audio analysis server 20, and a generation server 30. The video generation device 10, the audio analysis server 20, and the generation server 30 are connected to each other via a communication network NET so that they can communicate with one another.

[0012] Note that the fact that Figure 1 shows only one video generation device 10, one audio analysis server 20, and one generation server 30 is merely an example.

[0013] The video generation device 10 is a device for a user U[1] to obtain information about the highlight scene of the first video data MD1 based on the first video data MD1, and to extract the second video data MD2 from the first video data MD1 using this information. The second video data MD2 is, for example, video data showing the highlight scene of the first video data MD1. The video generation device 10 inputs a prompt PP containing multiple dialogue data LDs corresponding to multiple lines of dialogue contained in the first video data MD1 to a language model LM stored in the generation server 30, thereby obtaining specification information MI from the generation server 30 that specifies one or more dialogue data LDs corresponding to the highlight scene of the first video data MD1. The above prompt PP includes an instruction indicating that the highlight scene of the first video data MD1 is to be determined. This instruction is an example of a "first instruction".

[0014] The audio analysis server 20 receives the first video data MD1 from the video generation device 10 and transcribes the multiple lines of dialogue contained in the first video data MD1 to determine multiple first text data TD1 that correspond one-to-one with the lines of dialogue. The determined multiple first text data TD1 are transmitted from the audio analysis server 20 to the video generation device 10 along with the time information associated with the first text data TD1. When the audio analysis server 20 transcribes the above-mentioned multiple lines of dialogue, it is preferable to perform preprocessing such as removing noise and silent portions from the audio of the first video data MD1 and rounding the timestamps.

[0015] The generation server 30 receives the above prompt PP from the video generation device 10. The generation server 30 also stores a language model LM, which is a Large Language Model (LLM). One example of this language model LM is GPT-4o (registered trademark). The generation server 30 inputs the above prompt PP received from the video generation device 10 into the language model LM. The language model LM outputs information regarding the highlight scene of the first video data MD1. One example of this information is information indicating the location from which the second video data MD2, which shows the highlight scene of the first video data MD1, is extracted from the first video data MD1.

[0016] 1-1-2: Diagram 2 of the video generation device configuration is a block diagram showing an example configuration of the video generation device 10. As shown in Figure 2, the video generation device 10 comprises a processing unit 11, a storage device 12, a display device 13, an input device 14, and a communication device 15. Each element of the video generation device 10 is interconnected by one or more buses for communicating information.

[0017] The processing unit 11 is a processor that controls the entire video generation device 10. The processing unit 11 is configured using, for example, one or more chips. The processing unit 11 is configured using, for example, a central processing unit (CPU) that includes interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 11 may be implemented by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The processing unit 11 executes various processes in parallel or sequentially.

[0018] The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory).

[0019] The storage device 12 stores multiple programs, including the control program PR1, for execution by the processing device 11. The storage device 12 functions as a work area for the processing device 11. Furthermore, the storage device 12 may store, for example, the prompt PP described later.

[0020] The display device 13 is a device that displays images and text information. The display device 13 displays various images under the control of the processing device 11. For example, various display panels such as liquid crystal display panels and organic EL display panels are preferably used as the display device 13.

[0021] The input device 14 is a device that accepts operations from user U[1]. For example, the input device 14 is configured to include a keyboard, touchpad, touch panel, or pointing device such as a mouse. Here, if the input device 14 is configured to include a touch panel, it may also serve as the display device 13.

[0022] The communication device 15 is hardware that acts as a transmitting and receiving device for communicating with other devices. The communication device 15 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 15 may be equipped with a connector for wired connection and an interface circuit corresponding to the connector. The communication device 15 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include products compliant with wired LAN, IEEE1394, and USB. Examples of wireless communication interfaces include products compliant with wireless LAN and Bluetooth®.

[0023] The processing unit 11 functions as a communication control unit 111, a video acquisition unit 112, a text data acquisition unit 113, a determination unit 114, a prompt acquisition unit 115, an input unit 116, an information acquisition unit 117, a display control unit 118, and an extraction unit 119, for example, by reading and executing the control program PR1 from the storage device 12.

[0024] The communication control unit 111 causes the communication device 15 to send and receive various information, various data, and various signals between it and the voice analysis server 20 and the generation server 30. The communication control unit 111 may also cause the communication device 15 to send and receive various information, various data, and various signals between it and other devices on the communication network NET besides the voice analysis server 20 and the generation server 30.

[0025] The video acquisition unit 112 acquires the first video data MD1. The first video data MD1 may be video data input to the video generation device 10 by user U[1] using the input device 14. Alternatively, the first video data MD1 may be video data acquired from a device (not shown) located on the communication network NET.

[0026] The first video data MD1 acquired by the video acquisition unit 112 is transmitted to the voice analysis server 20 by the communication control unit 111. As described above, the voice analysis server 20 transcribes the multiple lines of dialogue contained in the first video data MD1 to determine a plurality of first text data TD1 that correspond one-to-one with the multiple lines of dialogue. The determined plurality of first text data TD1 are transmitted from the voice analysis server 20 to the video generation device 10 along with time information associated with each of the first text data TD1. This time information includes the start and end times of the dialogue corresponding to the first text data TD1.

[0027] Furthermore, the audio analysis server 20 may not only transcribe multiple lines of dialogue contained in the first video data MD1, but may also detect speaker information indicating the speaker of each line of dialogue. In this case, as described later, if this speaker information is additionally considered when extracting highlight scenes, the generation server 30 can improve the accuracy of extracting highlight scenes.

[0028] The text data acquisition unit 113 acquires multiple first text data TD1 transmitted from the speech analysis server 20.

[0029] The determination unit 114 determines a plurality of dialogue data LDs that correspond one-to-one to a plurality of dialogues by adding identification data DD, which identifies the dialogue, to each of the plurality of first text data TD1 acquired by the text data acquisition unit 113. Furthermore, the determination unit 114 determines a text data group TG that includes the plurality of first text data TD1 and the plurality of dialogue data LDs.

[0030] Figure 3 shows an example of a text data set TG.

[0031] The text data group TG has five items: "start", "end", "period", "text", and "llm_text". The "start" item stores data indicating the point in time when a line of dialogue begins to be spoken in the first video data MD1. The "end" item stores data indicating the point in time when the line of dialogue ends in the first video data MD1. The "period" item stores data indicating the duration of the line of dialogue in the first video data MD1. The "text" item stores data showing the transcription result of the line of dialogue. The first text data TD1 refers to the data showing the transcription result of the line of dialogue. The "llm_text" item stores dialogue data LD, which is the first text data TD1 stored in the "text" item with identification data DD added to identify the line of dialogue. In the example shown in Figure 3, the identification data DD is the sequential number in which the dialogue corresponding to the first text data TD1 occurs. The dialogue data LD is data in which the identification data DD that identifies the dialogue is attached to each of the multiple first text data TD1, and there is a one-to-one correspondence between multiple lines of dialogue.

[0032] In Figure 2, the prompt acquisition unit 115 acquires a prompt PP that includes the above-mentioned multiple dialogue data LDs.

[0033] Figure 4 shows an example of the prompt PP format. The prompt PP includes the first block BL1 to the fifth block BL5.

[0034] The first block BL1 includes an outline of instructions to the language model LM stored in the generation server 30. In the example shown in Figure 4, the first block BL1 includes instructions to determine the highlight scenes of the video content corresponding to the first video data MD1 and output keywords that symbolize the highlight scenes. More specifically, the first block BL1 includes instructions to determine three highlight scenes and output two keywords, as an example.

[0035] Block 2, BL2, contains the genre name of the video content and a transcript data file. The transcript data file contains a text data group TG. As described above, the text data group TG contains multiple dialogue data LDs.

[0036] Block 3, BL3, includes constraints for determining the highlight scenes. In the example shown in Figure 4, these constraints include conditions for the scenes to be extracted. These constraints also include an instruction to output specification information MI that specifies one or more dialogue data LDs corresponding to the highlight scenes by performing sentiment analysis on the dialogue indicated by the dialogue data LDs. In the example shown in Figure 4, it is instructed to use VADER®, a type of natural language processing, for the sentiment analysis. These constraints also include an instruction to output a scene title of 10 characters or less. These constraints also include an instruction that the dialogue data LDs corresponding to each highlight scene must be no more than five dialogue data LDs in sequence. These constraints also include constraints related to the genre specified in Block 2, BL2.

[0037] Block 4, BL4, includes the specification of the output format.

[0038] Block 5, BL5, includes examples of genre output.

[0039] Figure 5 is a diagram showing an example of the prompt PP. In the second block BL2 of the prompt PP shown in Figure 5, it is specified that the genre name is the live broadcast of sports. Also, in the third block BL3, as a constraint condition for determining the highlight scene, it is specified that the scene is the peak scene of the game. Furthermore, as the constraint condition, it is specified to prioritize the emphasis of the live broadcast and commentary, and the repeated scenes. Also, in the fifth block BL5, two output examples are specified.

[0040] In the above prompt PP, the instruction indicating that the highlight scene of the first video data MD1 is determined based on a plurality of line data LD is an example of the "first instruction". Also, the instruction indicating that the title of the highlight scene is output is an example of the "second instruction". Also, the instruction indicating that the specified information MI is output by performing sentiment analysis on the lines indicated by a plurality of line data LD is an example of the "third instruction". Also, the instruction indicating that a keyword symbolizing the highlight scene is output is an example of the "fourth instruction".

[0041] In Figure 2, the prompt acquisition unit 115 may acquire the prompt PP input by the user U[1] using the input device 14. Alternatively, the prompt acquisition unit 115 may acquire the prompt PP stored in the storage device 12.

[0042] The input unit 116 inputs the prompt PP acquired by the prompt acquisition unit 115 into the language model LM stored in the generation server 30.

[0043] The information acquisition unit 117 acquires the specified information MI that designates one or more line data LD corresponding to the highlight scene of the first video data MD1 output from the language model LM when the input unit 116 inputs the prompt PP into the language model LM.

[0044] Figure 6 is a diagram showing an example of the specified information MI. The specified information MI includes the sixth block BL6 to the eighth block BL8.

[0045] The sixth block BL6 includes the content of the first highlight scene, keywords, and the number of the line data LD included in the first highlight scene. The number is the identification data DD included in the line data LD and is the number in the order in which the lines occur.

[0046] The seventh block BL7 includes the content of the second highlight scene, keywords, and the number of the line data LD included in the second highlight scene.

[0047] The eighth block BL8 includes the content of the third highlight scene, keywords, and the number of the line data LD included in the third highlight scene.

[0048] In FIG. 2, the display control unit 118 causes the display device 13 to display the specified information MI acquired by the information acquisition unit 117.

[0049] The extraction unit 119 extracts second video data MD2 indicating the highlight scene of the first video data MD1 from the first video data MD1 based on the specified information MI acquired by the information acquisition unit 117. As an example, when the user U[1] manually extracts the second video data MD2 based on the specified information MI displayed on the display device 13, the extraction unit 119 extracts the second video data MD2 based on the operation information indicating the operation content of the user U[1] input to the input device 14. In this case, the user U[1] may refer to the text data group TG shown in FIG. 3 and use the "start" time point and "end" time point corresponding to the number of the line included in the specified information MI, that is, the identification data DD, to specify the extraction location from the first video data MD1.

[0050] As another example, the extraction unit 119 itself may extract the second video data MD2 based on the specified information MI acquired by the information acquisition unit 117. The extraction unit 119 may refer to the text data group TG shown in FIG. 3 and use the "start" time point and "end" time point corresponding to the number of the line included in the specified information MI, that is, the identification data DD, to specify the extraction location from the first video data MD1.

[0051] Figure 7 shows an example of the flow of various data and information used in the video generation device 10.

[0052] As described above with reference to Figure 2, the input unit 116 inputs the prompt PP to the language model LM stored in the generation server 30. The prompt PP includes a text data group TG. The text data group TG includes multiple dialogue data LDs. The multiple dialogue data LDs correspond one-to-one with multiple lines of dialogue. The multiple dialogue data LDs are determined by adding identification data DD, which identifies the lines of dialogue, to each of the multiple first text data TD1s, which are transcriptions of multiple lines of dialogue contained in the first video data MD1. Note that in Figure 7, for the sake of simplicity, the text data group TG and the prompt PP are depicted separately, but as described above with reference to Figures 4 and 5, the prompt PP includes the text data group TG.

[0053] The information acquisition unit 117 acquires the specified information MI output from the language model LM. The extraction unit 119 extracts the second video data MD2[1] corresponding to the first highlight scene, the second video data MD2[2] corresponding to the second highlight scene, and the second video data MD2[3] corresponding to the third highlight scene from the first video data MD1 based on the specified information MI.

[0054] 1-2: Diagram 8 of the operation of the first embodiment is a flowchart showing an example of the operation of the video generation device 10.

[0055] In step S1, the processing unit 11 of the video generation device 10 functions as a video acquisition unit 112. The processing unit 11 acquires the first video data MD1. The processing unit 11 also functions as a communication control unit 111. The processing unit 11 transmits the first video data MD1 to the audio analysis server 20.

[0056] In step S2, the processing unit 11 functions as a text data acquisition unit 113. The processing unit 11 acquires a plurality of first text data TD1 transmitted from the speech analysis server 20.

[0057] In step S3, the processing unit 11 functions as a determination unit 114. The processing unit 11 determines a plurality of dialogue data LDs that correspond one-to-one to a plurality of dialogues by adding identification data DDs that identify dialogues to each of the plurality of first text data TD1s acquired in step S2. Furthermore, the processing unit 11 determines a text data group TG that includes the plurality of first text data TD1s and the plurality of dialogue data LDs.

[0058] In step S4, the processing unit 11 functions as a prompt acquisition unit 115. The processing unit 11 acquires a prompt PP which includes multiple lines of dialogue data LD.

[0059] In step S5, the processing unit 11 functions as an input unit 116. The processing unit 11 inputs the prompt PP acquired in step S4 into the language model LM stored in the generation server 30.

[0060] In step S6, the processing unit 11 functions as an information acquisition unit 117. The processing unit 11 acquires specification information MI which specifies one or more dialogue data LDs from among the multiple dialogue data LDs output from the language model LM that correspond to the highlight scene of the first video data MD1.

[0061] In step S7, the processing unit 11 functions as a display control unit 118. The processing unit 11 displays the specified information MI acquired in step S6 on the display device 13.

[0062] In step S8, the processing unit 11 functions as an extraction unit 119. Based on the specified information MI acquired in step S6, the processing unit 11 extracts second video data MD2 from first video data MD1, which shows the highlight scene of first video data MD1.

[0063] 1-3: Effects of the First Embodiment The video generation device 10 according to this embodiment comprises a determination unit 114, an information acquisition unit 117, and an extraction unit 119. The determination unit 114 determines a plurality of dialogue data LDs that correspond one-to-one with a plurality of lines of dialogue contained in the first video data MD1, based on the first video data MD1. Each of the plurality of dialogue data LDs indicates at least the corresponding line of dialogue in text. The information acquisition unit 117 inputs a prompt PP containing the plurality of dialogue data LDs to the language model LM, and obtains designation information MI from the language model LM that designates one or more dialogue data LDs from the plurality of dialogue data LDs that correspond to the highlight scene of the first video data MD1. The extraction unit 119 extracts second video data MD2 that shows the highlight scene of the first video data MD1 from the first video data MD1 based on the designation information MI.

[0064] By having the above configuration, the video generation device 10 can handle a large amount of video processing with less effort compared to conventional technology when extracting highlight scenes from video data.

[0065] Furthermore, in conventional technology, the extraction of highlight scenes relied on the subjective judgment of the person in charge, resulting in a lack of objectivity and consistency in the selection and extraction results, and causing variability in the quality of the highlight scene videos. However, the video generation device 10 of this disclosure can significantly suppress subjective variability and improve the consistency of results by selecting highlight scenes based on objective analysis using LLM.

[0066] Furthermore, in the video generation device 10, each of the multiple dialogue data LDs includes a first text data TD1 that shows the corresponding dialogue in text and identification data DD that identifies the corresponding dialogue. The identification data DD indicates the order in which the dialogue occurs.

[0067] The video generation device 10, with the above configuration, can specify one or more dialogue data LDs corresponding to highlight scenes using the order in which the dialogue occurs.

[0068] Furthermore, in the video generation device 10, the prompt PP includes a first instruction that instructs the device to determine a highlight scene of the first video data MD1 based on a plurality of dialogue data LDs. The first instruction determines the portion of the first video data MD1 corresponding to a first number of consecutive lines of dialogue as the highlight scene. The designated information MI includes identification data DD that identifies the first number of consecutive lines of dialogue.

[0069] The video generation device 10, with the above configuration, can specify the highlight scene of the first video data MD1 using the number of lines of dialogue.

[0070] Furthermore, in the video generation device 10, prompt PP includes a second instruction indicating that the title of the highlight scene should be output. The information acquisition unit 117 acquires the title of the highlight scene.

[0071] The video generation device 10, with the above configuration, can add titles to highlight scenes to the second video data MD2.

[0072] Furthermore, in the video generation device 10, the prompt PP includes a third instruction that instructs the output of specified information MI by performing sentiment analysis on the lines indicated by multiple lines of dialogue data LD.

[0073] By having the above configuration, the video generation device 10 can determine the highlight scene of the first video data MD1 based on an emotional analysis of the dialogue.

[0074] Furthermore, in the video generation device 10, the prompt PP includes the genre of the first video data MD1 and constraints related to that genre. The information acquisition unit 117 acquires designated information MI that indicates a highlight scene of the first video data MD1 that satisfies the above constraints.

[0075] By having the above configuration, the video generation device 10 can determine highlight scenes that satisfy the genre-specific constraints of the first video data MD1.

[0076] Furthermore, in the video generation device 10, the prompt PP includes a fourth instruction that instructs the output of a keyword symbolizing the highlight scene. The information acquisition unit 117 acquires the keyword symbolizing the highlight scene.

[0077] The video generation device 10, with the above configuration, can add keywords for highlight scenes to the second video data MD2.

[0078] Furthermore, the video generation device 10 includes a display control unit 118 that causes the specified information MI to be displayed on the display device 13.

[0079] The video generation device 10, with the above configuration, allows user U[1] to visually inspect the specified information MI.

[0080] 2. Modifications The present disclosure is not limited to the embodiments illustrated above. Specific examples of modifications are given below.

[0081] 2-1: Modification 1 In the first embodiment, the video generation system 1 comprises a video generation device 10, an audio analysis server 20, and a generation server 30. However, in addition to these components, the video generation system 1 may also include a terminal 40 that is communicatively connected to the video generation device 10 via a communication network NET.

[0082] Figure 9 is a block diagram showing an example of the overall configuration of a video generation system 1A according to Modification 1. As shown in Figure 9, the video generation system 1A comprises a video generation device 10, an audio analysis server 20, a generation server 30, and terminals 40[1] to 40[j]. The video generation device 10, the audio analysis server 20, the generation server 30, and terminals 40[1] to 40[j] are connected to each other via a communication network NET, where j is an integer of 1 or more.

[0083] In Figure 9, user U uses terminal 40. Also, user U[1] uses terminal 40[1]. User U[2] uses terminal 40[2]. User U[k] uses terminal 40[k]. User U[j] uses terminal 40[j]. k is an integer between 1 and j, inclusive. In the following explanation, user U[k] will be used as a representative example of user U, and terminal 40[k] will be used as a representative example of terminal 40.

[0084] User U[k] may upload the first video data MD1 to the video generation device 10 using terminal 40[k]. Alternatively, user U[k] may upload the prompt PP to the video generation device 10 using terminal 40[k].

[0085] In this case, the display control unit 118 of the video generation device 10 causes the specified information MI to be displayed on a display device (not shown) provided on the terminal 40 [k].

[0086] When the first video data MD1 stored in terminal 40[k] is uploaded from terminal 40[k] to the video generation device 10, and the specified information MI is displayed on the display device of terminal 40[k], user U[k] can extract the second video data MD2 from the first video data MD1 stored in terminal 40[k].

[0087] 2-2: Modification 2 In the above embodiment, the language model LM stored in the generation server 30 receives a prompt PP, and the language model LM outputs specified information MI. The information acquisition unit 117 acquires the specified information MI. The extraction unit 119 extracts the second video data MD2 from the first video data MD1 based on the specified information MI.

[0088] However, the generation server 30 may store an image language model instead of the language model LM. In this case, the image language model may output the second video data MD2 instead of the specified information MI.

[0089] 2-3: Modification 3 In the above embodiment, the text data acquisition unit 113 acquires a plurality of first text data TD1 transmitted from the speech analysis server 20. Then, the determination unit 114 determines a plurality of dialogue data LD that correspond one-to-one to a plurality of dialogues by adding identification data DD that identifies the dialogue to each of the plurality of first text data TD1 acquired by the text data acquisition unit 113. Furthermore, the determination unit 114 determines a text data group TG that includes the plurality of first text data TD1 and the plurality of dialogue data LD.

[0090] However, the voice analysis server 20 may generate the text data group TG by having the same functions as the determination unit 114. In this case, the text data group TG is transmitted from the voice analysis server 20 to the video generation device 10.

[0091] 2-4: Modification 4 As described above, when the voice analysis server 20 detects speaker information indicating the speakers of multiple lines of dialogue, the determination unit 114 may include the speaker information in the dialogue data LD. In this case, the transcription data file included in the prompt PP also includes the speaker information. By including an instruction in the prompt PP to determine the highlight scene after additionally considering the speaker information, the generation server 30 can further improve the accuracy of extracting the highlight scene.

[0092] 2-5: Modification 5 In the above embodiment, the prompt PP includes a transcription data file, and the transcription data file includes a text data group TG. The prompt PP may further include a first video data MD1. In this case, by including an instruction in the prompt PP to determine a highlight scene, taking into additional consideration the information contained in the video indicated by the first video data MD1, the generation server 30 can further improve the accuracy of extracting the highlight scene. The information contained in the video includes, as an example, at least one of the characters contained in the video, the image contained in the video, the color tone of the video, and the movement of the video. Furthermore, this instruction is an example of a "fifth instruction".

[0093] 3. Other (1) In the embodiments described above, the storage device 12 is exemplified by ROM and RAM, but it can also be a flexible disk, magneto-optical disk (e.g., compact disk, digital multipurpose disk, Blu-ray® disk), smart card, flash memory device (e.g., card, stick, key drive), CD-ROM (Compact Disc-ROM), register, removable disk, hard disk, floppy® disk, magnetic strip, database, server, or other suitable storage medium.

[0094] (2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0095] (3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0096] (4) In the embodiments described above, the determination may be made by a value represented using one bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0097] (5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be rearranged in order, as long as there is no contradiction. For example, the methods described in this disclosure present various step elements using an exemplary order and are not limited to the specific order presented.

[0098] (6) Each function illustrated in Figures 1 to 9 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each function block is not particularly limited. That is, each function block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A function block may also be realized by combining the above one device or the above multiple devices with software.

[0099] (7) The programs illustrated in the embodiments described above should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages ​​or by other names.

[0100] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0101] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

[0102] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0103] (10) In the embodiments described above, the terminal 40 may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “mobile station,” “user terminal,” “user equipment (UE),” and “terminal” may be used interchangeably.

[0104] (11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain and optical (both visible and invisible) domain.

[0105] (12) In the embodiments described above, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on".

[0106] (13) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, or inquiring (e.g., searching in a table, database, or other data structure), or ascertaining. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), inputting, outputting, or accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0107] (14) In the embodiments described above, where “include,” “including,” and variations thereof are used, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0108] (15) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

[0109] (16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0110] (17) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of the specified information (e.g., notification that "it is X") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0111] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are illustrative and not restrictive in any way.

[0112] 1...Video generation system, 1A...Video generation system, 10...Video generation device, 11...Processing device, 12...Storage device, 13...Display device, 14...Input device, 15...Communication device, 20...Voice analysis server, 30...Generation server, 40...Terminal, 111...Communication control unit, 112...Video acquisition unit, 113...Text data acquisition unit, 114...Decision unit, 115...Prompt acquisition unit, 116...Input unit, 117...Information acquisition unit, 118...Display control unit, 119...Extraction unit, BL...Block, DD...Identification data, LD...Dialogue data, LM...Language model, MD1...First video data, MD2...Second video data, MI...Specified information, NET...Communication network, PP...Prompt, PR1...Control program, TD1...First text data, TG...Text data group, U...User

Claims

1. A video generation device comprising: a determination unit that determines a plurality of dialogue data that correspond one-to-one with a plurality of lines of dialogue contained in a first video data based on first video data, and each of the plurality of dialogue data indicates at least the corresponding line of dialogue in text; an information acquisition unit that inputs a prompt containing the plurality of dialogue data to a language model and acquires designation information from the language model that designates one or more lines of dialogue data from the plurality of dialogue data that correspond to a highlight scene of the first video data; and an extraction unit that extracts second video data showing a highlight scene of the first video data from the first video data based on the designation information.

2. Each of the plurality of dialogue data includes text data that shows the corresponding dialogue in text and identification data that identifies the corresponding dialogue, wherein the identification data indicates the order in which the dialogue occurs, the video generation device according to claim 1.

3. The video generation device according to claim 2, wherein the prompt includes a first instruction that instructs the device to determine a highlight scene of the first video data based on the plurality of dialogue data, the first instruction indicates that the portion of the first video data corresponding to a first number of consecutive lines of dialogue is to be determined as the highlight scene, and the designation information includes identification data that identifies the first number of consecutive lines of dialogue.

4. The video generation apparatus according to claim 1, wherein the prompt includes a second instruction indicating that the title of the highlight scene should be output, and the information acquisition unit acquires the title of the highlight scene.

5. The video generation device according to claim 2, wherein the prompt includes a third instruction that instructs the device to output the specified information by performing sentiment analysis on the dialogue indicated by the plurality of dialogue data.

6. The video generation apparatus according to claim 1, wherein the prompt includes the genre of the first video data and constraints relating to the genre, and the information acquisition unit acquires designation information indicating a highlight scene of the first video data that satisfies the constraints.

7. The video generation apparatus according to claim 1, wherein the prompt includes a fourth instruction that instructs the output of a keyword symbolizing the highlight scene, and the information acquisition unit acquires the keyword symbolizing the highlight scene.

8. The video generation apparatus according to claim 1, further comprising a display control unit that causes the specified information to be displayed on a display device.

9. The video generation device according to claim 1, wherein the plurality of dialogue data includes speaker information indicating the speaker of the dialogue shown by the plurality of dialogue data.

10. The video generating apparatus according to claim 1, wherein the prompt further includes a fifth instruction indicating that a highlight scene of the first video data is determined by additionally considering the first video data and the information contained in the video shown by the first video data.

Images