Video data analysis support system, information processing device, information processing method, and program

Abstract

A processing device 11 comprises: an object detection unit 12 that outputs, as frame information, position information and a class of an object detected using an object detection model for each frame of video data acquired by a video data acquisition unit 11, and extracts, from the frame information, frame information of target classes, which are classes of interest; an output conversion unit 13 that creates, for each target class, a class-specific position information group in which the position information is arranged in the order of frames in the video data; a prompt generation unit 14 that generates a prompt for each target class by adding the class-specific position information group to a question sentence requesting an explanation of the behavior of the target class; and a text information generation unit 15 that generates text information for each target class by inputting the prompt for the target class into a large-scale language model.

Description

Video data analysis support system, information processing device, information processing method, and program

[0001] The present invention relates to a video data analysis support system, information processing device, information processing method, and program that can easily and simply monitor the operation of a target being monitored in a video, and reduce monitoring errors.

[0002] On the production line, to improve the operational efficiency and safety of the factory, monitoring is carried out using IT and various sensors, and in areas where the application of various sensors is difficult, operator-based monitoring is also performed. Operator-based monitoring is carried out, for example, to grasp the progress of the production line and the operating status of equipment in real time, and to respond quickly when an abnormality occurs in order to improve production efficiency. Operators also monitor the status of equipment via monitors in order to improve maintenance efficiency and minimize downtime by detecting equipment failures and abnormalities early. Furthermore, remote monitoring is also carried out to centrally manage the status of multiple factories from a remote location. In addition to operator monitoring, camera images displayed on monitors are saved as recorded video data and played back and reviewed as needed.

[0003] Patent Document 1 discloses a method for classifying events using a text-video classification model that can classify events by specifying events with keywords written in text (for example, "staggering"). This method involves obtaining time-series probability data from video data, which shows the probability of the keyword being present at predetermined intervals in the video data, and determining whether or not a detection event to be detected exists from the time-series probability data.

[0004] In addition, Non-Patent Document 1 discloses a technology related to automatic generation of video captions, which focuses on generation of Japanese subtitles for describing human actions. A technology is disclosed for converting into text a description of the movement of an object in a video regarding "who did what where" using an encoder-decoder model that was constructed and learned from a large-scale Japanese video caption dataset consisting of videos and captions described in the format of "who did what where".

[0005] Japanese Patent No. 7409731

[0006] “Video Caption Dataset for Describing Human Actions in Japanese”, Yutaro Shigeto, Yuya Yoshikawa, Jiaqing Lin, Akikazu Takeuchi, Proceedings of The 12th Language Resources and Evaluation Conference(LREC), 2020, 4664-4670, [Retrieved on November 8, 2024], Internet <URL:https: / / arxiv.org / pdf / 2003.04865> “YOLO-World: Real-Time Open-Vocabulary Object Detection”, author [Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, Ying Shan], published conference [CVPR] 2024

[0007] By the way, monitor monitoring by an operator always requires monitoring the monitor to detect abnormalities. When performing video playback confirmation work of recorded camera images, it takes time and imposes a large burden, and there is a possibility that mistakes in the monitoring work may occur due to overlooking abnormalities. In particular, in the monitoring work of industrial equipment, it is required to accurately grasp the position and movement of the monitoring target without misunderstanding and quickly detect abnormalities.

[0008] To facilitate the automation of monitoring tasks and the consolidation of control rooms, it is conceivable to use the technology described in Non-Patent Document 1 to transcribe descriptions of the movement of objects in videos into text. However, the encoder-decoder model described in Non-Patent Document 1 requires the preparation of a massive training dataset in advance.

[0009] The present invention has been made in view of the above problems, and aims to provide a video data analysis support system, information processing device, information processing method, and program that can easily and simply monitor the movement of a target being monitored in a video, and reduce monitoring errors. Furthermore, it aims to shorten the time required for the monitor's decision-making and expedite anomaly detection. Conventional monitoring support systems only simply display video data, which has the problem of requiring time for anomaly detection and understanding of movement. The present invention combines an object detection model and a large-scale language model, and improves the processing efficiency of the computer by generating prompts with added coordinate information and displaying paths with axes, thereby achieving a technical improvement that expedites monitoring decisions.

[0010] The video data analysis support system according to the present invention is a video data analysis support system in which a computer assists in monitoring industrial equipment, and comprises: a video data acquisition unit that acquires video data consisting of multiple frames; an object detection unit that outputs the position information and class of an object detected using an object detection model as frame information for each frame of the video data, and extracts frame information of a target class from the frame information; an output conversion unit that creates a group of class-specific position information for each target class by arranging the position information in the order of the frames in the video data; a prompt generation unit that adds the group of class-specific position information to a question sentence requesting an explanation of the operation of the target class and generates a prompt for input to a large-scale language model for each target class; and a text information generation unit that inputs the prompt to a large-scale language model for each target class and generates text information using the large-scale language model.

[0011] The prompt generation unit may generate a prompt in which coordinate information relating to the correspondence between the coordinate system of the location information and the position on the screen displaying the video data is added to the question text.

[0012] The output conversion unit may convert the position information into a tabular format using Markdown notation.

[0013] The object detection unit may be provided with an input unit that inputs the target class and the prompt generation unit with the question text.

[0014] The system may also include an output unit that outputs the text information generated by the text information generation unit to support the monitoring of industrial equipment.

[0015] The output unit may graphically display and output the position information for each target class, along with the text information generated by the text information generation unit, as a path corresponding to the order of frames in the video data, based on a coordinate system having x and y axes and scales.

[0016] The information processing device according to the present invention is an information processing device in which a computer assists in the monitoring of industrial equipment, and is connected to an object detection unit that uses an object detection model to output object position information and class as frame information from video data composed of multiple frames, and a text information generation unit that uses a large-scale language model to generate text information based on input prompts, and is an information processing device that assists in the analysis of the video data, and comprises a video data acquisition unit that acquires the video data, a frame information acquisition unit that inputs the video data to the object detection unit and acquires frame information of a target class which is the target class, an output conversion unit that creates a class-specific position information group for each target class in which the position information is arranged in the order of the frames in the video data, a prompt generation unit that adds the class-specific position information group to a question sentence requesting an explanation of the operation of the target class and generates a prompt for input to a large-scale language model for each target class, and a text information acquisition unit that inputs the prompt to the text information generation unit for each target class and acquires the text information generated by the large-scale language model.

[0017] The prompt generation unit may generate a prompt by adding coordinate information relating to the correspondence between the coordinate system of the location information and the position on the screen displaying the video data to the question text, and output it to the text information generation unit.

[0018] The output conversion unit may convert the position information into a tabular format using Markdown notation.

[0019] The system may further include an input unit that inputs the target class to the object detection unit and the question text to the prompt generation unit.

[0020] The system may further include an output unit that outputs the text information acquired by the text information acquisition unit to support the monitoring of industrial equipment.

[0021] The output unit may graphically display and output the position information for each target class, along with the text information acquired by the text information acquisition unit, as a path corresponding to the order of frames in the video data, based on a coordinate system having x and y axes and scales.

[0022] The information processing method according to the present invention is an information processing method in which a computer assists in the monitoring of industrial equipment, and is connected to an object detection unit that uses an object detection model to output object position information and class as frame information from video data composed of multiple frames, and a text information generation unit that uses a large-scale language model to generate text information based on an input prompt, and is an information processing method that assists in the analysis of the video data, and includes: a video data acquisition step to acquire the video data; a frame information acquisition step to input the video data to the object detection unit and acquire frame information of a target class which is the target class; an output conversion step to create a class-specific position information group for each target class in which the position information is arranged in the order of the frames in the video data; a prompt generation step to add the class-specific position information group to a question sentence requesting an explanation of the operation of the target class and generate a prompt for input to a large-scale language model for each target class; and a text information acquisition step to input the prompt to the text information generation unit for each target class and acquire the text information generated by the large-scale language model.

[0023] The present invention relates to a program for an information processing device that is connected to an object detection unit that uses an object detection model to output object position information and class as frame information from video data composed of multiple frames, and a text information generation unit that uses a large-scale language model to generate text information based on an input prompt, and provides support for the analysis of the video data. The program causes the computer to execute the following steps: a video data acquisition step to acquire the video data; a frame information acquisition step to input the video data to the object detection unit and acquire frame information of a target class; an output conversion step to create a class-specific position information group for each target class, in which the position information is arranged in the order of the frames in the video data; a prompt generation step to generate a prompt for input to a large-scale language model for each target class by adding the class-specific position information group to a question sentence requesting an explanation of the operation of the target class; and a text information acquisition step to input the prompt to the text information generation unit for each target class and acquire the text information generated by the large-scale language model.

[0024] According to the present invention, the operation of a monitored object in a video can be easily and simply monitored, and monitoring errors can be reduced. In particular, according to the present invention, by generating prompts with coordinate information and displaying paths with axes, changes in the position of the monitored object can be grasped without misunderstanding, resulting in the technical effects of faster monitoring decisions and reduced misrecognition. Furthermore, in supporting the monitoring of industrial equipment, processing efficiency and judgment accuracy can be significantly improved compared to conventional monitoring systems.

[0025] Figure 1 is a functional block diagram showing the configuration of a video data analysis support system according to an embodiment of the present invention. Figure 2 is a flowchart showing the video data analysis support processing procedure by the processing device. Figure 3 is a diagram showing one of the frame images in a video taken from the side of a crane moving around in the manufacturing industry. Figure 4 is a diagram showing an example of a group of class-specific position information in which the cranes are arranged in the order of the frames. Figure 5 is a diagram showing the operation of the crane based on the group of class-specific position information shown in Figure 4. Figure 6 is a diagram showing an example of a prompt for the crane. Figure 7 is a diagram showing a coordinate system of position information with the upper left as the origin. Figure 8 is a diagram showing text information of the response result obtained by inputting the prompt shown in Figure 6 into a large-scale language model. Figure 9 is a diagram showing an example in which, along with the text information, the position information of the target class of object is plotted within the area of ​​the display screen, and the path of the object corresponding to the order of the frames in the video data is graphically displayed and output. Figure 10 is a functional block diagram showing the configuration of a video data analysis support system according to a modified example. Figure 11 is a flowchart showing the video data analysis support processing procedure by the information processing device.

[0026] The video data analysis support system, information processing device, information processing method, and program according to embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments, and the components in the following embodiments include those that are easily substituted or substantially identical to those that are easily substituted by those skilled in the art.

[0027] <Overall Configuration> Figure 1 is a functional block diagram showing the configuration of a video data analysis support system 1, which is an embodiment of the present invention. This video data analysis support system 1 performs processing to convert into text a description of the movement (change in position information) of an object (monitored object) in the video or video (video data) captured by the imaging device 3. The processing of the present invention includes a configuration in which video data acquired from the imaging device is executed on a processing device equipped with a CPU and a GPU and output to a display device in real time. This enables high-speed processing that makes use of hardware resources.

[0028] As shown in Figure 1, in the video data analysis support system 1, the processing unit 2 is connected to an imaging device 3, a video database 4, a terminal device 5, a display device 6, and a storage device 7. The processing unit 2 has a video data acquisition unit 11, an object detection unit 12, an output conversion unit 13, a prompt generation unit 14, a text information generation unit 15, and an output unit 16. The processing unit 2 stores programs corresponding to each of the functional units of the video data acquisition unit 11, object detection unit 12, output conversion unit 13, prompt generation unit 14, text information generation unit 15, and output unit 16 in a storage device such as non-volatile memory, loads these programs into memory, and executes them with the CPU to run the corresponding processes.

[0029] The imaging device 3 is an imaging device such as a camera or an industrial surveillance camera (ITV). The imaging device 3 captures an image of the area to be monitored and outputs the captured video data of the area to the video data acquisition unit 11. In other words, the video data acquisition unit 11 acquires the video data captured by the imaging device 3. The video data may be stored in a video database 4, and the video data acquisition unit 11 may acquire desired video data from the video database 4.

[0030] The object detection unit 12 outputs the position information and class of the object detected using the object detection model as frame information for each frame of the acquired video data, and extracts the frame information of the target class from the frame information.

[0031] The object detection unit 12 first outputs the class and position information of objects detected frame by frame using an object detection model for the acquired video data as frame information. The object detection model is generated by machine learning using known methods on training images in which the range of the object to be detected is enclosed by a rectangular frame (bounding box) in the image, and the category (class) of the object in each bounding box is associated with it. The object detection model consists of a deep neural network (DNN) that simultaneously estimates the position and classifies objects in an image. Examples of object detection models used include R-CNN (Regional CNN), YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and DETR (End-to-End Object Detection with Transformers).

[0032] The object detection unit 12 then extracts frame information of the target class, which is the class to be detected, from the output frame information. Specifically, the object detection unit 12 extracts frame information of the target class (object class, position information) based on keywords related to the target class input in the input unit 17. The keywords related to the target class may be stored in advance in the processing unit 2. This extraction function is implemented as a function within the object detection model (YOLO-World) described in Non-Patent Literature 2, but it may also have a separate function unit from the object detection model.

[0033] The input unit 17 accepts a keyword related to the class (target class), which is an attribute of the target object, if the target object to be detected is predetermined. By entering a keyword related to the target class, the object detection model in the object detection unit 12 can limit the object detection process to only objects of the target class, thereby reducing the subsequent processing load. A class refers to an attribute of the target object, such as a category like human, building, or car.

[0034] The output conversion unit 13 creates a class-specific location information group for each target class, arranging the location information in the order of the frames in the video data. In other words, the output conversion unit 13 groups each output frame by object (class) and creates a class-specific location information group by arranging the location information for each target object in the order of the frames in the video data.

[0035] The prompt generation unit 14 adds class-specific location information to question sentences requesting explanations about the operation of the target class, and generates prompts for inputting large language models (LLMs) for each target class.

[0036] In other words, the prompt generation unit 14 adds a phrase (question) specifying an arbitrary output content (method of answer) that requests an explanation about the operation of the target class to the group of class-specific position information, which is the output processed by the output conversion unit 13, and creates a prompt to input to the large-scale language model for each target class. The question may be stored in advance, or it may be input from the input unit 17.

[0037] The text information generation unit 15 inputs the prompts created by the prompt generation unit 14 into a large-scale language model for each target class, and generates text information using this large-scale language model.

[0038] In other words, the text information generation unit 15 outputs text information describing the movement of an object of the target class by inputting a prompt into the large-scale language model. The large-scale language model is a language model constructed using a large amount of data and deep learning technology, and is modeled using the occurrence probabilities of sentences and words. It is a well-known computer language model used for text generation, translation, question answering, etc. As the large-scale language model, for example, a pre-trained base model of Llama3:70b (70 billion parameters, pre-trained version) can be used. Llama3:70b is open source, the randomness parameter can be fixed, and it has high accuracy, but it is not limited to this, and a self-made model or other large-scale language model may also be used.

[0039] The output unit 16 outputs the text information generated by the text information generation unit 15 to the display device 6, displaying the change in the position of the monitored object (i.e., its operation) as text information. The output unit 16 may also output the generated text information to the storage device 7 and store it as a text log.

[0040] <Video Data Analysis Support Processing> Figure 2 is a flowchart showing the video data analysis support processing procedure by the processing unit 2. As shown in Figure 2, the processing unit 2 first has the video data acquisition unit 11 acquire video data consisting of multiple frames (step S101). Then, for each frame of the video data, the object detection unit 12 outputs the position information and class of the object detected using the object detection model as frame information, and extracts the frame information of the target class, which is the target class, from the frame information (step S102).

[0041] Subsequently, the output conversion unit 13 creates a class-specific location information group for each target class, arranging the location information in the order of frames in the video data (step S103). Furthermore, the prompt generation unit 14 adds the class-specific location information group to a question sentence requesting an explanation of the operation of the target class, generating a prompt for input to the large-scale language model for each target class (step S104). Then, the text information generation unit 15 inputs the generated prompt to the large-scale language model for each target class, and the large-scale language model generates text information (step S105). Finally, the output unit 16 outputs the text information generated by the text information generation unit 15 (step S106), and this process ends.

[0042] <An Example of Video Data Analysis Support Processing> Next, as an example of video data analysis support processing, we will explain its application to monitoring the movement of a crane 20 in the manufacturing industry. Figure 3 shows one frame image from a video taken from the side of a crane 20 moving around in the manufacturing industry. In other words, Figure 3 shows an example of the position detection results of "crane" and "slab" in one frame image of the video data. In Figure 3, the object detection model detects the crane 20 and other slabs, including the slab 30, using bounding boxes in the image of the crane 20 transporting the slab 30. In Figure 3, the object name and confidence level are displayed in the upper right tag of each bounding box. Furthermore, frames are extracted from the video data at predetermined time intervals, for example, every second, and object detection processing is performed on the extracted frames.

[0043] The object detection model used is YOLO-World, as described above, but it is not limited to this model; any model that detects objects and outputs the object's position (for example, the x,y coordinates of the top-left corner of the bounding box) and object name will suffice. YOLO-World performs object detection for specified keywords without training, so the input unit 17 receives "crane" and "slab" as target classes, and the object detection unit 12 outputs the position information for the crane and slab.

[0044] The output of the object detection model lists the detected objects and their positions according to each frame (the progression of time in the video). Therefore, the data is grouped into "cranes" and multiple "slabs," and for each class, position information (class-specific position information group D1) is output in the order of the frames. Figure 4 shows an example of class-specific position information group D1 with "cranes" arranged in the order of the frames. In Figure 4, the class-specific position information group D1 for the crane class (object class) is shown, and the position information (x, y) for each crane arranged in the order of the frames (0, 7, 11, 14, 17) is shown in a table format. The order of the frames is the order in which the video data is composed, and they are not consecutive because the frames for object detection are selected at predetermined time intervals, for example, every second.

[0045] When there are multiple identical objects, they are added to the object class and assigned an ID with a number, and grouping is performed for the identical objects including the assigned ID. In the object detection result, confidence is included, but since this processing is unnecessary, it is deleted. Also, as position information, the positions of the four corners of the bounding box (top-left: top-left, bottom-right: bottom-right) are output, but only the top-left coordinate position of the bounding box may be acquired and the bottom-right coordinate position may be deleted. Also, objects with low confidence in the object detection result, for example, objects with a confidence of less than 0.4, may be deleted from the detection target. The position information is based on the top-left coordinates of the bounding box, and scale information is added as needed. Frames are extracted at a predetermined time interval (e.g., 1 second), and if the confidence of the object detection result is less than the threshold value (e.g., 0.4), they are excluded. By extracting frames at a predetermined time interval and excluding detection results with a confidence less than the threshold value in this way, the amount of analysis target data can be reduced, the CPU load and memory usage can be reduced, and efficient use of computer resources can be realized.

[0046] FIG. 5 is a diagram showing the operation of the crane based on the class-based position information group D1 shown in FIG. 4. As shown in FIG. 5, based on the class-based position information group D1, it is possible to know the position change and path of the crane over time in the video display area. In the present embodiment, at least the content shown in FIG. 5 is output as text information.

[0047] For this reason, the prompt generation unit 14 generates a prompt D10 for input to the large language model shown in FIG. 6. FIG. 6 is a diagram showing an example of the prompt D10 for the crane. In the prompt shown in FIG. 6, the class-based position information group D1 of the crane shown in FIG. 4 is converted into a table format using markdown notation.

[0048] That is, in the prompt D10, as the class-based position information group D1 of the crane, - coordinate data - | frame | x | y | | - | - | - | | 0 | 1280 | 720 | | 7 | 510 | 90 | | 11 | 70 | 450 | | 14 | 400 | 720 | | 17 | 730 | 450 | is described.

[0049] As the prompt question text, including the way of answering, the following text is generated: "You are an expert in coordinate data interpretation. This is the data detected by YOLOv7. It is extracted from a video under the following conditions. 1 frame: 1 second, Resolution: 1280×720. Briefly describe the details of the movement in text based on the coordinate data. - Response conditions - ・Japanese (English is not allowed). ・No extra preambles are required." This question text is input in advance from the input unit 17.

[0050] Coordinate information D2 is added to the prompt D10 as position information. That is, - Coordinate information - Resolution: 1280x720 (x,y)=(0,0): top - left (x,y)=(1280,720): bottom - right (x,y)=(1280,0): top - right (x,y)=(0,720): bottom - left is added. This coordinate information D2 is coordinate information regarding the correspondence between the coordinate system (x, y) of the position information and the position on the screen where the video data is displayed, that is, it is the one with the addition of the coordinate direction expression. Fig. 7 is a diagram showing the coordinate system of the position information with the upper left as the origin. Coordinate information D2 designates the coordinate system shown in Fig. 7. By specifying this coordinate information D2, an answer of text information D20 corresponding to the specified coordinate system is obtained, and the coordinate system when the monitor reads the text information D20 is specified, and troublesome operations such as subsequent coordinate system conversions can be reduced. The coordinate information can include the resolution of the video data (e.g., 1280×720), the origin position (upper left), the axis directions (the x - axis is the horizontal direction, the y - axis is the vertical direction), and scale information or graduations as necessary. Thereby, the explanation generated by the large - scale language model conforms to the coordinate system of the monitoring screen, and the risk that the monitor misunderstands the positional relationship can be reduced.

[0051] Figure 8 shows the text information D20 obtained as a response result when prompt D10 shown in Figure 6 is input to a large-scale language model. Specifically, the text information obtained is: "According to this coordinate data, the crane in the video exhibits the following motion. First, in frame 0, the crane is in the lower right corner of the screen (x=1280, y=720). In frame 7, it moves towards the upper left, reaching the position x=510, y=90. In frame 11, it moves further towards the lower left, reaching the position x=70, y=450. Then, in frame 14, it moves towards the lower right, reaching the position x=400, y=720. Finally, in frame 17, it moves towards the upper right, reaching the position x=730, y=450. In other words, this crane moves within the screen in the order of upper left → lower left → lower right → upper right." The content shown in Figure 5 is returned as text information.

[0052] When a prompt for a single slab 30 is generated in the same way as for crane 20 and input into a large-scale language model, the text information of the response is obtained as follows: "According to this coordinate data, the slab in the video exhibits the following motion. First, in frame 0, the slab is in the lower right corner of the screen (x=1280, y=720). In frame 7, it moves towards the upper left, reaching the position x=510, y=90. In frame 11, it moves further towards the lower left, reaching the position x=70, y=450. Then, in frame 14, it moves towards the lower right, reaching the position x=400, y=720. Finally, in frame 17, it moves towards the upper right, reaching the position x=730, y=450. In other words, this slab moves across the screen in the order of upper left → lower left → lower right → upper right."

[0053] Furthermore, by combining these answers and making a judgment, if the "crane" and the "slab" are moving in the same way, it can be determined that the crane is gripping and moving the slab.

[0054] Furthermore, when these results are used to input a prompt into a large-scale language model asking for an explanation of the movement and relationship between the crane and the slab, the following response is obtained: "Since the crane and the slab are moving in the same order and to the same position, it can be inferred that the crane is probably lifting or moving the slab. In other words, it is suggested that the crane is manipulating the slab." This provides a function as a video data analysis support system that automatically explains the content of the video.

[0055] By the way, in the above embodiment, text information D20 was output as the response result, but it is not limited to this, and as shown in Figure 9, along with the text information D20 generated by the text information generation unit 15, the position information of the target class object may be plotted in the display screen area E corresponding to Figure 5, and the path of the object corresponding to the order of frames in the video data may be graphically displayed and output. This graphical display allows the content of the response in text information D20 to be visually recognized. This graphical display may be displayed in real time. Furthermore, the output unit can display the position information of the target class as a path based on a coordinate system having x and y axes and scales, along with the text information. This display allows the monitor to intuitively grasp the direction and speed of movement of the target object, enabling quick anomaly detection and operational judgment. This makes it easy and simple to monitor the movement of the target object in the video and reduces monitoring errors.

[0056] <Modification> In the embodiment described above, the processing unit 2 performed the main video data analysis processing, and the imaging device 3, video database 4, terminal device 5, display device 6, and storage device 7 were directly connected to the processing unit 2. However, regardless of the imaging device 3, video database 4, terminal device 5, display device 6, and storage device 7, each functional unit within the processing unit 2 may also be distributed via a network or the like. In particular, the object detection unit 12 that uses an object detection model and the text information generation unit 15 that uses a large-scale language model are expected to be used in a configuration where they are connected externally.

[0057] Figure 10 is a functional block diagram showing the configuration of a modified video data analysis support system 101. The video data analysis support system 101 shown in Figure 10 has the functions of the object detection unit 12 and the text information generation unit 15 in the information processing device 102, which corresponds to the processing device 2, externally connected via a network N. The object detection unit 12 performs object detection using the object detection model M1, similar to the object detection unit 12 of the processing device 2. The text information generation unit 15 outputs text information using the large-scale language model M2, similar to the text information generation unit 15 of the processing device. The terminal device 105 has an input unit 17, similar to the terminal device 5, but also has an output unit 16 that corresponds to the output unit 16 of the processing device 2.

[0058] In other words, as shown in Figure 10, the information processing device 102, terminal device 105, video database 4, imaging device 3, object detection unit 12, and text information generation unit 15 are interconnected via a network N, and the display device 6 and storage device 7 are connected to the output unit 16. The information processing device 102, object detection unit 12, text information generation unit 15, etc., may be built on the cloud.

[0059] The information processing device 102 includes a frame information acquisition unit 112 for obtaining detection results by having the object detection unit 12 perform object detection via the network N, and a text information acquisition unit 115 for obtaining text information by having the text information generation unit 15 perform output processing of text information via the network N. Furthermore, like the processing device 2, the information processing device 102 also includes a video data acquisition unit 11, an output conversion unit 13, and a prompt generation unit 14.

[0060] The frame information acquisition unit 112 inputs the video data acquired from the video data acquisition unit 11 to the object detection unit 12 and acquires frame information for the target class. The text information acquisition unit 115 inputs the prompts generated by the prompt generation unit 14 to the text information generation unit 15 for each target class and acquires text information generated by the large-scale language model M2. The frame information acquisition unit 112 may also have the frame information extraction function that the object detection unit 12 of this embodiment has.

[0061] The information processing device 102 stores programs corresponding to each of its functional units—the video data acquisition unit 11, the frame information acquisition unit 112, the output conversion unit 13, the prompt generation unit 14, and the text information acquisition unit 115—in a storage device such as non-volatile memory. By loading these programs into memory and executing them with the CPU, the corresponding processes are executed.

[0062] <Video Data Analysis Support Processing by Information Processing Device> Figure 11 is a flowchart showing the video data analysis support processing procedure by the information processing device 102. As shown in Figure 11, the information processing device 102 first has a video data acquisition unit 11 acquire video data consisting of multiple frames (step S201). Then, the frame information acquisition unit 112 sends the video data to the object detection unit 12, which outputs the position information and class of the object detected using the object detection model M1 as frame information for each frame of the video data, and the frame information of the extracted target class is acquired from the frame information (step S202).

[0063] Subsequently, the output conversion unit 13 creates a class-specific location information group for each target class, arranging the location information in the order of frames in the video data (step S203). Furthermore, the prompt generation unit 14 adds the class-specific location information group to a question sentence requesting an explanation of the operation of the target class, generating a prompt for input to the large-scale language model for each target class (step S204). Then, the text information acquisition unit 115 sends the generated prompt to the text information generation unit 15 for each target class, causing the text information generation unit 15 to generate text information using the large-scale language model M2, and acquires the generated text information (step S205). Finally, the information processing device 102 outputs the text information generated by the text information generation unit 15 via the output unit 16 (step S206), and terminates this process.

[0064] In the above embodiment, processing was performed on a single video data, but multiple video data may be processed simultaneously in parallel to centralize monitoring. In the modified example, each information processing device 102 can share the object detection unit 12 and the text information generation unit 15, resulting in a simpler system configuration. The above video data analysis support processing may be performed in real time, or it may be stored in the video database 4 first and then processed afterward. Furthermore, although the output unit displayed text information, it may also output text information as audio information.

[0065] According to the above embodiments and modifications, by combining a general-purpose object detection model and a general-purpose large-scale language model, the actions of the monitored object in the video can be output as text information, thereby enabling easy and convenient motion monitoring and reducing monitoring errors.

[0066] The video data analysis support system, information processing device, information processing method, and program according to the present invention have been specifically described above with reference to embodiments and modifications for carrying out the invention. However, the spirit of the present invention is not limited to these descriptions and must be interpreted broadly based on the claims. Furthermore, it goes without saying that various changes and modifications based on these descriptions are also included in the spirit of the present invention.

[0067] According to the present invention, it is possible to provide a video data analysis support system, an information processing device, an information processing method, and a program that can easily and simply monitor the operation of a target being monitored in a video, and reduce monitoring errors.

[0068] 1,101 Video data analysis support system 2 Processing unit 3 Imaging device 4 Video database 5,105 Terminal device 6 Display device 7 Storage device 11 Video data acquisition unit 12 Object detection unit 13 Output conversion unit 14 Prompt generation unit 15 Text information generation unit 16 Output unit 17 Input unit 20 Crane 30 Slab 102 Information processing unit 112 Frame information acquisition unit 115 Text information acquisition unit D1 Class-specific position information group D2 Coordinate information D10 Prompt D20 Text information E Area M1 Object detection model M2 Large-scale language model N Network

Claims

1. A video data analysis support system that uses a computer to assist in monitoring industrial equipment, comprising: a video data acquisition unit that acquires video data consisting of multiple frames; an object detection unit that outputs the position information and class of an object detected using an object detection model as frame information for each frame of the video data, and extracts frame information of a target class from the frame information; an output conversion unit that creates a group of class-specific position information for each target class by arranging the position information in the order of the frames in the video data; a prompt generation unit that adds the group of class-specific position information to a question sentence requesting an explanation of the operation of the target class, and generates a prompt for input to a large-scale language model for each target class; and a text information generation unit that inputs the prompt to a large-scale language model for each target class and generates text information using the large-scale language model.

2. The video data analysis support system according to claim 1, wherein the prompt generation unit generates a prompt that adds coordinate information relating to the correspondence between the coordinate system of the position information and the position on the screen displaying the video data to the question sentence.

3. The video data analysis support system according to claim 1 or 2, wherein the output conversion unit converts the position information into a tabular format using Markdown notation.

4. The video data analysis support system according to any one of claims 1 to 3, further comprising an input unit that inputs the target class to the object detection unit and the question text to the prompt generation unit.

5. The video data analysis support system according to any one of claims 1 to 4, further comprising an output unit that outputs text information generated by the text information generation unit for the purpose of supporting the monitoring of industrial equipment.

6. The video data analysis support system according to claim 5, wherein the output unit graphically displays and outputs the position information for each target class, along with the text information generated by the text information generation unit, as a path corresponding to the order of frames in the video data, based on a coordinate system having x and y axes and scales.

7. An information processing device that supports the monitoring of industrial equipment by a computer, comprising: an object detection unit that uses an object detection model to output object location information and class as frame information from video data composed of multiple frames; and a text information generation unit that uses a large-scale language model to generate text information based on input prompts, and which is connected to the object detection unit to support the analysis of the video data, comprising: a video data acquisition unit that acquires the video data; a frame information acquisition unit that inputs the video data to the object detection unit and acquires frame information of a target class which is the target class; an output conversion unit that creates a class-specific location information group for each target class in which the location information is arranged in the order of the frames in the video data; a prompt generation unit that adds the class-specific location information group to a question sentence requesting an explanation of the operation of the target class and generates a prompt for input to a large-scale language model for each target class; and a text information acquisition unit that inputs the prompt to the text information generation unit for each target class and acquires text information generated by the large-scale language model.

8. The information processing apparatus according to claim 7, wherein the prompt generation unit generates a prompt by adding coordinate information relating to the correspondence between the coordinate system of the position information and the position on the screen displaying the video data to the question sentence and outputs it to the text information generation unit.

9. The information processing apparatus according to claim 7 or 8, wherein the output conversion unit converts the position information into a tabular format using Markdown notation.

10. The information processing apparatus according to any one of claims 7 to 9, further comprising an input unit that inputs the target class to the object detection unit and the question text to the prompt generation unit.

11. The information processing apparatus according to any one of claims 7 to 10, further comprising an output unit that outputs text information acquired by the text information acquisition unit for the purpose of supporting the monitoring of industrial equipment.

12. The information processing apparatus according to claim 11, wherein the output unit graphically displays and outputs the position information for each target class, together with the text information acquired by the text information acquisition unit, as a path corresponding to the order of frames in the video data, based on a coordinate system having x and y axes and scales.

13. An information processing method for which a computer supports the monitoring of industrial equipment, comprising: an object detection unit that uses an object detection model to output object location information and class as frame information from video data composed of multiple frames; and a text information generation unit that uses a large-scale language model to generate text information based on input prompts, and an information processing method for which the computer supports the analysis of the video data, comprising: a video data acquisition step to acquire the video data; a frame information acquisition step to input the video data to the object detection unit and acquire frame information of a target class that is the target class; an output conversion step to create a class-specific location information group for each target class in which the location information is arranged in the order of the frames in the video data; a prompt generation step to add the class-specific location information group to a question sentence requesting an explanation of the operation of the target class and generate a prompt for input to a large-scale language model for each target class; and a text information acquisition step to input the prompt to the text information generation unit for each target class and acquire text information generated by the large-scale language model.

14. A program for an information processing device that supports the analysis of video data, connected to an object detection unit that uses an object detection model to output object position information and class as frame information from video data composed of multiple frames, and a text information generation unit that uses a large-scale language model to generate text information based on input prompts, the program causing a computer to execute: a video data acquisition procedure to acquire the video data; a frame information acquisition procedure to input the video data to the object detection unit and acquire frame information of a target class that is the target class; an output conversion procedure to create a class-specific position information group for each target class, in which the position information is arranged in the order of the frames in the video data; a prompt generation procedure to add the class-specific position information group to a question sentence requesting an explanation of the operation of the target class and generate a prompt for input to a large-scale language model for each target class; and a text information acquisition procedure to input the prompt to the text information generation unit for each target class and acquire the text information generated by the large-scale language model.

Images