Video processing system, video processing method, and program

Abstract

The present invention is capable of determining the degree of similarity between images according to the contents of video. This video processing system comprises: a similarity estimation network for estimating, through a neural network, a first similarity index value indicating the degree of similarity between a first frame image and a second frame image that are included in a video; an image language model that outputs, to an image to be input, an answer obtained by verbalizing the image to be input; a similarity calculation unit that calculates a second similarity index value indicating the degree of similarity between a first answer obtained by inputting the first frame image to the image language model and a second answer obtained by inputting the second frame image to the image language model; and a training unit that trains the similarity estimation network so as to reduce the difference between the first similarity index value and the second similarity index value.

Description

Image processing system, image processing method, and program

[0001] The present disclosure relates to an image processing system, an image processing method, and a program.

[0002] As related art, Patent Document 1 discloses a machine learning system. In the machine learning system described in Patent Document 1, the first model generates an image vector representing the features of the input image and generates a sentence corresponding to the image. The second model generates an image vector representing the features of the input image and generates a sentence corresponding to the image. The machine learning device inputs a learning image into the first model and generates an image vector of the learning image. Further, the machine learning device inputs the correct corresponding sentence of the learning image into the first model and generates the corresponding sentence of the learning image. Furthermore, the machine learning device inputs a sentence with correct answers, for which it is known whether it corresponds to the learning image, into the second model and generates a sentence vector of the sentence with correct answers. The machine learning device updates the parameters of each of the first model and the second model so that the error between the correct corresponding sentence and the generated corresponding sentence, and the error between the similarity between the image vector and the sentence vector and the correct answer converge.

[0003] Japanese Patent Application Laid-Open No. 2023-56798

[0004] However, when attempting to apply a technique that requires input of text and images, such as the prior art, in the field of image processing, in order to perform specific processing based on an image, a text related to the image is required, and there is a possibility that the image processing cannot be performed efficiently.

[0005] One object of the present disclosure is to provide an image processing system, an image processing method, and a program that enable determination of the similarity between images according to the content of an image.

[0006] A video processing system according to a first aspect of the present disclosure includes: a similarity estimation network that estimates a first similarity index value indicating the similarity between a first frame image and a second frame image included in a video using a neural network; an image language model that outputs a response that verbalizes the input image for an input image; a similarity calculation unit that calculates a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image to the image language model and a second response obtained by inputting the second frame image to the image language model; and a learning unit that learns the similarity estimation network so that the difference between the first similarity index value and the second similarity index value becomes small.

[0007] A second aspect of the present disclosure of a video processing method includes: estimating a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; calculating a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of the input image, and a second response obtained by inputting the second frame image into the image language model; and training the neural network to reduce the difference between the first similarity index value and the second similarity index value.

[0008] A program according to a third aspect of this disclosure estimates a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network, calculates a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of the input image, and a second response obtained by inputting the second frame image into the image language model, and causes a computer to perform a process of training the neural network so that the difference between the first similarity index value and the second similarity index value becomes small.

[0009] The video processing system, video processing method, and program related to this disclosure enable the determination of similarity between images according to the content of the video.

[0010] This is a block diagram showing a schematic configuration example of the video processing system related to this disclosure. This is a block diagram showing a configuration example of the video processing system related to this disclosure. This is a block diagram showing a configuration example of the learning system. This is a diagram schematically showing the calculation of the measured similarity score. This is a diagram schematically showing video splitting. This is a flowchart showing the operation procedure of the learning system. This is a flowchart showing the operation procedure of the video splitting device. This is a block diagram showing a configuration example of the computer device.

[0011] Prior to describing the embodiments of this disclosure, the inventors will explain the matters they have considered. Consider a system in which a user searches for desired video from video collected from multiple vehicles. In such a system, consider that a long video, for example, one hour long, is divided into multiple video clips, and each divided video clip is compressed with appropriate compression parameters. Also consider that video search is performed on the divided video clips. In that case, if the video is simply divided at fixed time intervals, efficient compression and appropriate video search may be difficult. If the video content of similar parts of the video were to be divided into a single video clip manually, the amount of work would be enormous. In the field of video processing, there is a need for a mechanism that can determine the degree of similarity between images in order to divide video into scenes, search for images based on images, and compress similar images.

[0012] This document outlines the contents of this disclosure. Figure 1 is a block diagram showing an example of a schematic configuration of the image processing system related to this disclosure. The image processing system 10 includes a similarity estimation network 11, an image language model 12, a similarity calculation unit 13, and a learning unit 14.

[0013] The similarity estimation network 11 estimates a first similarity index value, which indicates the similarity between a first frame image and a second frame image contained in the video, using a neural network. The image language model 12 is a model that outputs a verbalized response to an input image. Here, the image language model can mean a model that handles images and text in combination. The "verbalized response to the image" can mean text generated from the input image through multimodal processing of images and language. The verbalized response to the image can also be expressed as a response that describes the image in text. The image language model 12 outputs a first response to the first frame image. The image language model 12 outputs a second response to the second frame image.

[0014] The similarity calculation unit 13 calculates a second similarity index value that indicates the similarity between the first response and the second response output from the image language model 12. The learning unit 14 trains the similarity estimation network 11 so that the difference between the first similarity index value and the second similarity index value becomes small.

[0015] In this embodiment, the similarity calculation unit 13 calculates a second similarity index value based on the first and second responses obtained by inputting the first and second frame images into the image language model 12. The learning unit 14 learns the similarity estimation network 11 based on the first similarity index value estimated by the similarity estimation network 11 and the second similarity index value calculated by the similarity calculation unit 13. In this way, a similarity estimation network 11 can be obtained that outputs a high similarity index value for two frame images whose responses obtained by the image language model 12 are similar. By using such a similarity estimation network 11, it is possible to determine the similarity between images according to the content of the video. By generating a model that takes images as input, acquired video can be input into the model, and video processing can be performed efficiently.

[0016] The embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. Furthermore, in the following drawings, the same elements and similar elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0017] Figure 2 is a block diagram showing an example configuration of the video processing system according to this disclosure. In Figure 2, the video processing system is configured as a video search system 100. An embodiment will be described using Figure 2. The video search system 100 shown in Figure 2 includes a compression device 110 and a video search device 130.

[0018] The video search system 100 can be used, for example, to search for desired video footage from video footage collected from multiple vehicles. For example, multiple vehicles each transmit driving video footage, which is video footage taken using cameras mounted on the vehicles, to a video collection device (not shown) via a wireless network. The compression device 110 encodes the video using a predetermined encoding scheme and stores the encoded video in the video storage unit 150. In this embodiment, video encoding is considered synonymous with video compression. The video storage unit 150 uses high-speed storage such as a solid-state drive (SSD). The video search device 130 searches for desired video footage from the video footage stored in the video storage unit 150 in response to a prompt entered by the user. Here, the prompt can mean an instruction or search query entered by the user.

[0019] The video retrieval device 130 includes a decoder 131 and a recognition unit 132. Physically, the video retrieval device 130 may be configured as a device having one or more memories and one or more processors. In the video retrieval device 130, at least some of the functions of each part within the video retrieval device 130 can be realized by one or more processors executing processing according to instructions read from one or more memories. The decoder 131 may be configured as a dedicated hardware circuit.

[0020] The decoder 131 decompresses or decodes the video compressed by the compression device 110. The search prompt 133 is a prompt entered by the user during a video search. The search prompt 133 is written in natural language, for example. The search prompt 133 includes words that specify objects and situations to be recognized by the recognition unit 132. The recognition unit 132 retrieves the video stored in the video storage unit 150 via the decoder 131 and generates recognition results for the search prompt 133 based on the retrieved video. Based on the recognition results, the recognition unit 132 extracts the video corresponding to the objects and situations specified in the search prompt 133 from the video storage unit 150. The recognition unit 132 includes an artificial intelligence model, such as a Vision Language Model (VLM).

[0021] The compression device 110 includes a video splitting device 111 and an encoder 116. The video splitting device 111 includes an image input unit 112, a similarity estimation network 113, a determination unit 114, and a splitting unit 115. Physically, the compression device 110 and the video splitting device 111 may each be configured as devices having one or more memories and one or more processors. In the compression device 110 and the video splitting device 111, at least some of the functions of each part within the compression device 110 and the video splitting device 111 can be realized by one or more processors executing processing according to instructions read from one or more memories. The encoder 116 may be configured by a dedicated hardware circuit.

[0022] The image input unit 112 inputs two consecutive frame images from the video 200 to be divided into the similarity estimation network 113. Of the two images input into the similarity estimation network 113, the image with the smaller frame number is also called the first frame image, and the image that follows the first frame image is also called the second frame image. The frame number is related to the time of the frame image in the video, and a smaller frame number means that the time is earlier. The image input unit 112 sequentially acquires pairs of consecutive frame images from the video 200, shifting the frame number by one each time, and sequentially inputs the acquired pairs of frame images into the similarity estimation network 113.

[0023] The similarity estimation network 113 estimates the similarity between two input images. The similarity estimation network 113 includes a neural network. The similarity estimation network 113 takes two frame images as input to the neural network and outputs a similarity score indicating the similarity between the two frame images. The similarity score corresponds to a similarity index value, that is, an index value indicating similarity. The similarity score estimated by the similarity estimation network 113 is also called the estimated similarity score or the first similarity index value.

[0024] The decision unit 114 determines whether or not to divide the video 200 based on the estimated similarity score output from the similarity estimation network 113. For example, if the estimated similarity score is less than a predetermined threshold, the decision unit 114 decides to divide the video 200 between two frame images input to the similarity estimation network 113. The division unit 115 divides the video 200 at the frame that the decision unit 114 has decided to divide the video 200 at. The resulting partial video is also called a video clip. The division unit 115 divides the video 200 into a video clip with the frame of the first frame image as the final frame and a video clip with the frame of the second frame image as the first frame.

[0025] A minimum sequence length may be set for the video clip. For example, the decision unit 114 may decide to split the video if the estimated similarity score is less than a predetermined threshold and the sequence length of the video clips after splitting is greater than or equal to the predetermined sequence length. The decision unit 114 will not split the video 200 if, even if the estimated similarity score is less than a predetermined threshold, the sequence length of the video clips after splitting would be less than the predetermined sequence length.

[0026] The encoder 116 encodes the video clips that have been divided by the video splitting device 111 while compressing them. The encoder 116 is also called the video compression unit. The encoder 116 may encode the video clips using video parameters set according to the video clips to be compressed. The video parameters include, for example, the video resolution, frame rate, and bitrate. The encoder 116 may also set a focus area within the frame image of the video clip and compress the frame image so that the image quality of the focus area is relatively higher than the image quality of other areas. The encoder 116 stores the compressed video clips in the video storage unit 150. The encoder 116 may distribute the compressed and encoded video clips to the video retrieval device 130 via the network.

[0027] In the example shown in Figure 2, the video splitting device 111 is included in the compression device 110, but the video splitting device 111 does not necessarily have to be included in the compression device 110. The video splitting device 111 may be configured as a device independent of the compression device 110.

[0028] Figure 3 is a block diagram showing an example configuration of a learning system used to generate an estimation model 310. The learning system 300 includes a frame image acquisition unit 301, a similarity estimation network 302, an image language model 303, a similarity calculation unit 304, and a learning unit 305. Physically, the learning system 300 can be configured as a device having one or more memories and one or more processors. In the learning system 300, at least some of the functions of each part within the learning system 300 can be realized by one or more processors executing processing according to instructions read from one or more memories. The learning system 300 does not need to be configured as a separate system from the video search system 100 shown in Figure 2, and some elements of the video search system 100 may overlap with some elements of the learning system 300. The learning system 300 corresponds to the video processing system 10 shown in Figure 1.

[0029] The frame image acquisition unit 301 acquires two frame images from the training video 330. For example, the frame image acquisition unit 301 acquires a frame image of an arbitrary frame number and a frame image of a frame consecutive to that frame image from the training video 330. The two frame images acquired by the frame image acquisition unit 301 are also called the first frame image and the second frame image.

[0030] The similarity estimation network 302 includes, for example, a neural network such as a Convolutional Neural Network (CNN), Transformer, or Multi-layer Perceptron (MLP), or a combination thereof. The similarity estimation network 302 uses the neural network to estimate an estimated similarity score 360 ​​that indicates the similarity between two frame images acquired by the frame image acquisition unit 301. The similarity estimation network 302 corresponds to the similarity estimation network 11 shown in Figure 1.

[0031] The image language model 303 generates a response that verbalizes the input image in response to the input image. The image language model 303 includes, for example, a large-scale language model generated by learning from samples. The image language model 303 generates a response in response to the input prompt. The response generated by the image language model 303 is, for example, written in a predetermined language. The image language model 303 may be the same as the image language model used for recognition processing in the recognition unit 132 included in the video retrieval device 130. The image language model 303 corresponds to the image language model 12 shown in Figure 1.

[0032] The learning prompts 350 include a plurality of prompts used to train the similarity estimation network 302. The learning prompts 350 may include a plurality of prompts that are assumed to be search prompts 133 input to the recognition unit 132 in the video retrieval device 130. The learning prompts 350 may be generated in response to the training video 330, for example, using Generative Artificial Intelligence (AI). The learning prompts 350 include a variety of prompts or complex prompts that may be considered for the training video 330. For example, the image language model 303 generates responses for each of the N prompts included in the learning prompts 350 for each of two frame images, where N is an integer of 2 or more.

[0033] The similarity calculation unit 304 calculates an actual similarity score 370 that indicates the similarity between the responses to two frame images generated by the image language model 303. The similarity calculation unit 304 calculates an actual similarity score 370 that indicates the similarity between the first response generated by the image language model 303 for the first frame image and the second response generated by the image language model 303 for the second frame image. The actual similarity score 370 is also called the second similarity index value. The similarity calculation unit 304 calculates the similarity score between the first response and the second response using, for example, an evaluation framework such as G-Eval. The similarity calculation unit 304 may calculate the similarity score between the first response and the second response for each of the multiple learning prompts 350, and calculate the average of the calculated similarity scores as the actual similarity score 370. The similarity calculation unit 304 corresponds to the similarity calculation unit 13 shown in Figure 1.

[0034] The learning unit 305 learns the similarity estimation network 302 based on the estimated similarity score 360 ​​and the measured similarity score 370. The learning unit 305 learns the similarity estimation network 302 so that the difference between the estimated similarity score 360 ​​and the measured similarity score 370 decreases. The learned similarity estimation network 302 is used as the similarity estimation network 113 in the video splitting device 111 shown in Figure 2. The learning unit 305 corresponds to the learning unit 14 shown in Figure 1.

[0035] The calculation of the measured similarity score 370 will now be explained. Figure 4 is a schematic diagram showing the calculation of the measured similarity score. The frame image acquisition unit 301 acquires two consecutive frame images, 331 and 332, from the training video 330. The image language model 303 generates responses 380-1 to 380-N (responses 1-1 to 1-N) for each of the N prompts 350-1 to 350-N (prompts 1 to N) included in the training prompts 350 for frame image 331. The image language model 303 also generates responses 390-1 to 390-N (responses 2-1 to 2-N) for each of the N prompts 350-1 to 350-N included in the training prompts 350 for frame image 332.

[0036] The similarity calculation unit 304 calculates similarity scores for each prompt between the responses 380-1 to 380-N for frame image 331 and the responses 390-1 to 390-N for frame image 332. In other words, the similarity calculation unit 304 calculates similarity scores 400-1 to 400-N (similarity score 1 to similarity score N) for each of the N prompts 350-1 to 350-N. Based on the similarity scores 400-1 to 400-N, the similarity calculation unit 304 calculates the measured similarity score 370. For example, the similarity calculation unit 304 calculates the measured similarity score 370 as the arithmetic mean of the similarity scores 400-1 to 400-N.

[0037] Next, we will explain video splitting based on estimated similarity scores. Figure 5 is a schematic diagram of video splitting. In this example, the minimum sequence length of the split video clips is assumed to be "4". That is, the video clips are split so that they contain at least four frame images.

[0038] As shown in Figure 5, suppose the estimated similarity score generated by the similarity estimation network 113 for frame image 201 and frame image 202 falls below a threshold. In that case, the decision unit 114 determines the space between the frame of frame image 201 and the frame of frame image 202 as the division point D1. The division unit 115 divides the portion of the video from the video 200 up to the division point D1 into a video clip. In other words, the division unit 115 extracts a video clip from the video 200 consisting of frames from the first frame of video 200, or the frame corresponding to the previous division point, up to the frame of frame image 201.

[0039] Suppose the estimated similarity score generated by the similarity estimation network 113 for frame image 203 and frame image 204 falls below a threshold. The decision unit 114 checks the number of frames between the frame of frame image 203 and the frame of frame image 204 from the previous division point D1. In the example in Figure 5, the number of frames between the frame of frame image 203 and the frame of frame image 204 from the previous division point D1 is "2". If the video 200 were to be divided between the frame of frame image 203 and the frame of frame image 204, the video clip with the frame of the divided frame image 202 as the first frame would not meet the minimum sequence length requirement. In this case, the decision unit 114 decides not to divide the video, even if the estimated similarity score is below the threshold.

[0040] Next, suppose the estimated similarity score generated by the similarity estimation network 113 for frame image 205 and frame image 206 falls below a threshold. The decision unit 114 checks the number of frames between the frame of frame image 205 and the frame of frame image 206 from the previous division point D1. In the example in Figure 5, the number of frames between the frame of frame image 203 and the frame of frame image 204 from the previous division point D1 is "4". In this case, the decision unit 114 determines the area between the frame of frame image 205 and the frame of frame image 206 as division point D2. The division unit 115 divides the video 200 into video clips, specifically the portion of the video from division point D1 to division point D2. In other words, the division unit 115 extracts video clips from the video 200 that consist of frame images 202 to 205.

[0041] Next, the operation procedure will be explained. Figure 6 is a flowchart of the operation procedure of the learning system 300. The operation procedure of the learning system 300 corresponds to the video processing method. The frame image acquisition unit 301 acquires two frame images from the learning video 330 (step A1). For example, the frame image acquisition unit 301 acquires the frame image of frame i and the frame image of frame i+1, with i being an arbitrary integer randomly selected.

[0042] The similarity estimation network 302 estimates an estimated similarity score 360 ​​between the two frame images acquired in step A1 (step A2). The image language model 303 generates responses for each of the two frame images selected in step A1 (step A3). The similarity calculation unit 304 calculates an actual similarity score 370 indicating the similarity of the responses for the two frame images generated in step A3 (step A4). Note that steps A2 and steps A3 and A4 may be performed in any order. Alternatively, steps A2 and steps A3 and A4 may be performed in parallel.

[0043] The learning unit 305 learns the similarity estimation network 302 based on the estimated similarity score 360 and the measured similarity score 370 (step A5). The learning system 300 learns the similarity estimation network 302 by repeatedly performing steps A1 to A9 on a plurality of learning videos 330. The learning system 300 may repeatedly perform steps A1 to A9 while changing the frame images obtained from the learning video 330 for one learning video 330. The learned similarity estimation network 302 is used as the similarity estimation network 113 in the video segmentation device 111.

[0044] FIG. 7 is a flowchart showing the operation procedure of the video segmentation device 111. The image input unit 112 inputs two consecutive frame images from the video 200 to be segmented into the similarity estimation network 113 (step B1). In step B1, the image input unit 112 sequentially inputs pairs of two consecutive frame images to the similarity estimation network 113 while shifting the frame numbers one by one. The similarity estimation network 113 estimates the similarity score between the two input frame images (step B2).

[0045] The determination unit 114 determines whether or not the estimated similarity score is less than the threshold value (step B3). When the determination unit 114 determines that the similarity score is less than the threshold value, if the video is segmented into video clips, the determination unit 114 determines whether or not the sequence length of the video clip is equal to or greater than the minimum sequence length (step B4). If it is determined in step B3 that the similarity score is not less than the threshold value, the process returns to step B1, and the pair of frame images is input to the similarity estimation network 113. If it is determined in step B4 that the sequence length is not equal to or greater than the minimum sequence length, the process also returns to step B1, and the pair of frame images is input to the similarity estimation network 113.

[0046] When the determination unit 114 determines in step B4 that the sequence length is equal to or greater than the minimum sequence length, it determines to divide the video between the two frame images acquired in step B1. In that case, the division unit 115 divides the video 200 between the two frame images acquired in step B1 (step B5). Thereafter, the process returns to step B1, and a pair of frame images is input to the similarity estimation network 113.

[0047] In the present embodiment, in the learning system 300, the similarity calculation unit 304 calculates an actual similarity score 370 based on the answers for two frame images obtained by the image language model 303. The learning unit 305 learns the similarity estimation network 302 based on the estimated similarity score 360 estimated by the similarity estimation network 302 and the actual similarity score 370. By doing so, it is possible to obtain a similarity estimation network that outputs a high similarity score for two frame images for which the answers obtained by the image language model 303 are similar. By using such a similarity estimation network 302, it is possible to determine the similarity between images according to the content of the video.

[0048] In the present embodiment, the similarity estimation network 302 is learned so that the difference between the estimated similarity score 360 and the actual similarity score 370 decreases. The learned similarity estimation network 302 is used as the similarity estimation network 113. In this case, in the video division device 111, the similarity estimation network 113 can output a high estimated similarity score for two frame images for which similar answers are obtained for the prompt in the image language model. In the video division device 111, by dividing the video when the similarity score estimated by the similarity estimation network 113 is low, it is possible to include in one video clip a video for which an answer similar to the prompt assumed in video search is obtained. It is considered that the video search device 130 can perform video search on such video clips and improve the quality of video search.

[0049] In video segmentation, if the sequence length of the segmented video clips is too short, noise included in the recognition results of the recognition process performed on the video clips makes it difficult to estimate appropriate compression parameters considering the quality of the recognition results, thus reducing the quality of video search. Conversely, if the sequence length of the segmented video clips is too long, changes occur in the video content within the video clips, making it difficult to determine appropriate compression parameters uniformly considering the quality of the recognition results. Therefore, when compressed with uniform compression parameters, the encoding efficiency is likely to decrease. In this embodiment, the video can be segmented according to the similarity score estimated using the trained similarity estimation network 113, and the video 200 can be segmented into video clips of appropriate lengths according to the video situation. By appropriately setting video parameters for each of the segmented video clips and performing image compression, the video clips can be compressed efficiently.

[0050] In the above embodiment, an example was described in which compressed video stored in the video storage unit 150 is used for video retrieval, but this disclosure is not limited thereto. Video clips that are divided using the similarity score estimated by the similarity estimation network 113 in the video splitting device 111 can be used not only for video retrieval but also for other purposes.

[0051] Next, the physical configuration of the compression device 110, the video splitting device 111, and the learning system 300 will be described. Figure 8 is a block diagram showing an example configuration of a computer device that can be used as the compression device 110, the video splitting device 111, or the learning system 300. The computer device 500 has a processor 510 such as a Central Processing Unit (CPU), a storage unit 520, a Read Only Memory (ROM) 530, a Random Access Memory (RAM) 540, a communication interface (IF) 550, and a user interface 560.

[0052] The communication interface 550 is an interface for connecting the computer device 500 to a communication network via wired communication means or wireless communication means. The user interface 560 includes a display unit, such as a display. The user interface 560 also includes input units such as a keyboard, mouse, and touch panel.

[0053] The memory unit 520 is an auxiliary storage device capable of holding various types of data. The memory unit 520 does not necessarily have to be part of the computer device 500; it may be an external storage device or cloud storage connected to the computer device 500 via a network.

[0054] ROM 530 is a non-volatile memory device. For example, a semiconductor memory device such as a relatively small-capacity flash memory is used for ROM 530. The program executed by the CPU 510 can be stored in the storage unit 520 or ROM 530. The storage unit 520 or ROM 530 stores various programs that realize the functions of the compression device 110, the video splitting device 111, or each part of the learning system 300.

[0055] The above program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, SSD or other memory technologies, Compact Disc (CD), digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0056] The RAM 540 is a volatile memory device. Various semiconductor memory devices such as Dynamic Random Access Memory (DRAM) or Static Random Access Memory (SRAM) can be used for the RAM 540. The RAM 540 may be used as an internal buffer for temporarily storing data. The CPU 510 loads a program stored in the memory unit 520 or ROM 530 into the RAM 540 and executes it. By executing the program, the CPU 510 can realize the functions of the compression device 110, the video splitting device 111, or each part of the learning system 300. The CPU 510 may have an internal buffer that can temporarily store data.

[0057] In the above embodiment, the compression device 110 and the video splitting device 111 do not necessarily have to be configured as a single computer device. The compression device 110 and the video splitting device 111 may each be configured using multiple physically separated devices. Similarly, the learning system 300 may also be configured using multiple physically separated devices.

[0058] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0059] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0060] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0061] [Note 1] A video processing system comprising: a similarity estimation network that estimates a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; an image language model that outputs a response that verbalizes the input image for an input image; a similarity calculation unit that calculates a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into the image language model and a second response obtained by inputting the second frame image into the image language model; and a learning unit that trains the similarity estimation network so that the difference between the first similarity index value and the second similarity index value becomes small.

[0062] [Note 2] The image language model is a model that generates the response in response to an input prompt, as described in Note 1, for the video processing system.

[0063] [Note 3] The video processing system according to Note 2, wherein the similarity calculation unit calculates the similarity between the first answer and the second answer for each of the multiple prompts, and calculates the average of the calculated similarities as the second similarity index value.

[0064] [Note 4] The video processing system according to any one of Notes 1 to 3, wherein the second frame image is an image of a frame that is continuous with the frame of the first frame image in the video.

[0065] [Note 5] The video processing system according to any one of Notes 1 to 4, further comprising: an image input unit that sequentially inputs two consecutive frame images from the video to be divided as the first frame image and the second frame image to the similarity estimation network; a decision unit that determines whether or not to divide the video to be divided based on the first similarity index value estimated by the similarity estimation network; and a division unit that divides the video to be divided into video clips at the frames in which the decision unit has decided to divide the video.

[0066] [Note 6] The video processing system according to Note 5, wherein the determination unit determines that if the first similarity index value is less than a predetermined threshold, the video to be divided is divided between two frame images input to the similarity estimation network.

[0067] [Note 7] The video processing system according to Note 5, wherein the determination unit determines to divide the video to be divided between two frame images input to the similarity estimation network when the first similarity index value is less than a predetermined threshold and the sequence length of the divided video is greater than or equal to a predetermined sequence length.

[0068] [Appendix 8] The video processing system according to any one of the appendices 5 to 7, further comprising a video compression unit that compresses the video to be divided for each of the divided video clips.

[0069] [Note 9] A video processing method comprising: estimating a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; calculating a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of the input image, and a second response obtained by inputting the second frame image into the image language model; and training the neural network to minimize the difference between the first similarity index value and the second similarity index value.

[0070] [Note 10] A program that causes a computer to perform the following processes: estimate a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; calculate a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of the input image, and a second response obtained by inputting the second frame image into the image language model; and train the neural network so that the difference between the first similarity index value and the second similarity index value becomes small.

[0071] Some or all of the elements (e.g., configuration and function) described in Appendices 2 through 8 that are dependent on Appendice 1 may also be dependent on Appendices 9 and 10 in the same way as those described in Appendices 2 through 8. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.

[0072] This application claims priority based on Japanese Patent Application No. 2024-206331, filed on 27 November 2024, and incorporates all of its disclosures herein.

[0073] 10: Video processing system 11: Similarity estimation network 12: Image language model 13: Similarity calculation unit 14: Learning unit 100: Video search system 110: Compression device 111: Video splitting device 112: Image input unit 113: Similarity estimation network 114: Decision unit 115: Splitting unit 116: Encoder 130: Video search device 131: Decoder 132: Recognition unit 133: Search prompt 150: Video storage unit 200: Video 201 to 206: Frame images 300: Learning system 301: Frame image acquisition unit 302: Similarity estimation network 303: Image language model 304: Similarity calculation unit 305: Learning unit 330: Video for learning 331, 332: Frame images 350: Prompts for learning 350-1 to 350-N: Prompts 370: Measured similarity score; 380-1 to 380-N, 390-1 to 390-N: Answers; 400-1 to 400-N: Similarity score; 500: Computer device; 510: Processor; 520: Memory unit; 530: ROM; 540: RAM; 550: Communication interface; 560: User interface

Claims

1. A video processing system comprising: a similarity estimation network that estimates a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; an image language model that outputs a response that verbalizes the input image for an input image; a similarity calculation unit that calculates a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into the image language model and a second response obtained by inputting the second frame image into the image language model; and a learning unit that trains the similarity estimation network so that the difference between the first similarity index value and the second similarity index value becomes small.

2. The image language model is a model that generates the response in response to an input prompt, according to claim 1.

3. The video processing system according to claim 2, wherein the similarity calculation unit calculates the similarity between the first answer and the second answer for each of the plurality of prompts, and calculates the average of the calculated similarities as the second similarity index value.

4. The video processing system according to any one of claims 1 to 3, wherein the second frame image is an image of a frame that is continuous with the frame of the first frame image in the video.

5. The video processing system according to any one of claims 1 to 4, further comprising: an image input unit that sequentially inputs two consecutive frame images from a video to be divided as the first frame image and the second frame image to the similarity estimation network; a decision unit that determines whether or not to divide the video to be divided based on the first similarity index value estimated by the similarity estimation network; and a division unit that divides the video to be divided into video clips at the frames in which the decision unit has determined to divide the video.

6. The video processing system according to claim 5, wherein the determination unit determines to divide the video to be divided between two frame images input to the similarity estimation network if the first similarity index value is less than a predetermined threshold.

7. The video processing system according to claim 5, wherein the determination unit determines to divide the video to be divided between two frame images input to the similarity estimation network if the first similarity index value is less than a predetermined threshold and the sequence length of the divided video is greater than or equal to a predetermined sequence length.

8. The video processing system according to any one of claims 5 to 7, further comprising a video compression unit that compresses the video to be divided into each divided video clip.

9. A video processing method comprising: estimating a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; calculating a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of an input image, and a second response obtained by inputting the second frame image into the image language model; and training the neural network to minimize the difference between the first similarity index value and the second similarity index value.

10. A program that causes a computer to perform the following processes: estimate a first similarity index value indicating the similarity between a first frame image and a second frame image contained in a video using a neural network; calculate a second similarity index value indicating the similarity between a first response obtained by inputting the first frame image into an image language model that outputs a verbalized response of an input image, and a second response obtained by inputting the second frame image into the image language model; and train the neural network so that the difference between the first similarity index value and the second similarity index value becomes small.

Images