Video processing apparatus and video processing method

Abstract

This video processing apparatus comprises: a candidate procedure group acquisition unit (1) for acquiring a plurality of candidate procedure groups, each candidate procedure group including a plurality of procedures that can possibly be a series of procedures for executing a task that appears in a moving image, and a procedure name of each of the plurality of procedures; and an action section detection unit (2) for detecting, on the basis of a change over time in a feature amount of the task that appears in the moving image, an action section in which each of the series of procedures for executing the task may appear in the moving image. The video processing apparatus further comprises a candidate procedure group selection unit (3) for selecting, from among the plurality of candidate procedure groups acquired by the candidate procedure group acquisition unit (1), a candidate procedure group that corresponds to the action section detected by the action section detection unit (2).

Description

Image processing device and image processing method

[0001] This disclosure relates to an image processing device and an image processing method.

[0002] There is a video processing device that detects action segments within a video, which are segments in which each of the steps for performing a task shown in the video is shown. As such a video processing device, for example, Non-Patent Literature 1 discloses a video processing device comprising a feature extraction unit that extracts feature quantities of a task shown in a video and outputs time-series data showing the change in feature quantities over time, and a clustering unit that detects action segments in which each of the steps is shown based on the change in feature quantities shown in the time-series data output from the feature extraction unit.

[0003] Kumar, Sateesh, et al. "Unsupervised action segmentation by joint representation learning and online clustering." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0004] The video processing device disclosed in Non-Patent Document 1 can detect action segments in which each step of a series of procedures is shown, but it has the problem of not being able to obtain the procedure name of the procedure corresponding to the action segment. As a result, even if a user looks at the detection results of the video processing device, they may not be able to easily understand what procedure corresponds to the action segment.

[0005] This disclosure was made to solve the above-mentioned problems and aims to provide a video processing device that can obtain the procedure name of a procedure corresponding to an action interval.

[0006] The video processing device according to this disclosure includes a procedure group candidate acquisition unit that acquires multiple procedure group candidates, each including multiple possible steps for executing a task shown in a video and the name of each step in those steps; and an action interval detection unit that detects action intervals in a video, which are sections in a video in which each of the steps for executing a task may be shown, based on the change in the feature quantities of the task shown in the video over time. The video processing device also includes a procedure group candidate selection unit that selects a procedure group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit that corresponds to an action interval detected by the action interval detection unit.

[0007] According to this disclosure, the procedure name of the procedure corresponding to the action interval can be obtained.

[0008] This is a configuration diagram showing an image processing device according to Embodiment 1. This is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 1. This is a hardware configuration diagram of a computer when the image processing device is implemented by software or firmware, etc. This is a flowchart showing an image processing method, which is the processing procedure of the image processing device. This is an explanatory diagram showing an example of three procedure group candidates created by generation AI. This is an explanatory diagram showing an example of multiple interval group candidates being output from a neural model such as a TAS model. This is an action send date graph showing an example of interval group candidate selection by the procedure group candidate selection processing unit 3b. This is an explanatory diagram showing an example of action intervals and procedure names presented by the presentation processing unit 4. This is a configuration diagram showing an image processing device according to Embodiment 2. This is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 2. This is a configuration diagram showing an image processing device according to Embodiment 3. This is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 3. This is a flowchart showing an image processing method, which is the processing procedure of the image processing device.

[0009] To provide a more detailed explanation of this disclosure, the forms for implementing this disclosure will be described below with reference to the attached drawings.

[0010] Embodiment 1. Figure 1 is a configuration diagram showing an image processing device according to Embodiment 1. Figure 2 is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 1. The image processing device shown in Figure 1 comprises a procedure group candidate acquisition unit 1, an action interval detection unit 2, a procedure group candidate selection unit 3, and a presentation processing unit 4.

[0011] The procedure group candidate acquisition unit 1 is implemented, for example, by the procedure group candidate acquisition circuit 21 shown in Figure 2. The procedure group candidate acquisition unit 1 acquires multiple procedure group candidates that are different from each other. The procedure group candidate includes multiple possible steps for a series of steps to perform the task shown in the video, and the names of each step in those steps. The procedure group candidate acquisition unit 1 outputs information indicating the multiple procedure group candidates to the procedure group candidate selection unit 3.

[0012] The action interval detection unit 2 is implemented, for example, by the action interval detection circuit 22 shown in Figure 2. The action interval detection unit 2 includes a feature extraction unit 2a and an interval group candidate acquisition unit 2b. Based on the changes in the feature quantities of the task shown in the video over time, the action interval detection unit 2 detects action intervals in the video, which are intervals in the video that may contain each of the steps in a series of steps for executing the task. The action interval detection unit 2 outputs the detection results of the action intervals to the step group candidate selection unit 3.

[0013] The feature extraction unit 2a acquires a video in which the task is shown. The feature extraction unit 2a extracts the features of the task shown in the video. The feature extraction unit 2a outputs time-series data showing the change in features over time to the interval group candidate acquisition unit 2b.

[0014] The interval group candidate acquisition unit 2b acquires time-series data from the feature extraction unit 2a. Based on the time changes of the features shown in the time-series data, the interval group candidate acquisition unit 2b acquires multiple distinct interval group candidates. Each interval group candidate includes an action interval that may represent each of the steps in a sequence of steps for performing a task. As a result of detecting action intervals, the interval group candidate acquisition unit 2b outputs information indicating the multiple interval group candidates to the procedure group candidate selection unit 3.

[0015] The procedure group candidate selection unit 3 is implemented, for example, by the procedure group candidate selection circuit 23 shown in Figure 2. The procedure group candidate selection unit 3 includes a matching processing unit 3a and a procedure group candidate selection processing unit 3b. The procedure group candidate selection unit 3 acquires information indicating a plurality of procedure group candidates from the procedure group candidate acquisition unit 1 and acquires the detection result of an action interval from the action interval detection unit 2. From the plurality of procedure group candidates acquired by the procedure group candidate acquisition unit 1, the procedure group candidate selection unit 3 selects a procedure group candidate that corresponds to the action interval detected by the action interval detection unit 2. The procedure group candidate selection unit 3 outputs information indicating the selected procedure group candidate to the presentation processing unit 4.

[0016] The matching processing unit 3a obtains information indicating multiple procedure group candidates from the procedure group candidate acquisition unit 1 and information indicating multiple interval group candidates from the interval group candidate acquisition unit 2b. The matching processing unit 3a compares each procedure group candidate with each interval group candidate. The matching processing unit 3a outputs the comparison result between each procedure group candidate and each interval group candidate to the procedure group candidate selection processing unit 3b.

[0017] The procedure group candidate selection processing unit 3b acquires information indicating multiple procedure group candidates from the procedure group candidate acquisition unit 1 and information indicating multiple interval group candidates from the interval group candidate acquisition unit 2b. The procedure group candidate selection processing unit 3b also acquires the matching result from the matching processing unit 3a. Based on the matching result from the matching processing unit 3a, the procedure group candidate selection processing unit 3b selects an interval group candidate from among the multiple interval group candidates acquired by the interval group candidate acquisition unit 2b that contains an action interval in which each of the steps for executing the task is shown. Based on the matching result from the matching processing unit 3a, the procedure group candidate selection processing unit 3b selects a procedure group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit 1 that corresponds to the action interval included in the selected interval group candidate. The procedure group candidate selection processing unit 3b outputs the information indicating the selected interval group candidate and the information indicating the selected procedure group candidate to the presentation processing unit 4.

[0018] The presentation processing unit 4 is implemented, for example, by the presentation processing circuit 24 shown in Figure 2. The presentation processing unit 4 obtains information indicating a group of interval candidates and information indicating a group of procedure candidates from the procedure group candidate selection processing unit 3b. The presentation processing unit 4 presents the action intervals included in the group of interval candidates selected by the procedure group candidate selection processing unit 3b, and the procedure names of the procedures included in the group of procedure candidates selected by the procedure group candidate selection processing unit 3b. Specifically, the presentation processing unit 4 displays the action intervals and procedure names on a display device, for example, not shown.

[0019] In Figure 1, it is assumed that the components of the video processing device—the procedure group candidate acquisition unit 1, the action interval detection unit 2, the procedure group candidate selection unit 3, and the presentation processing unit 4—are each implemented by dedicated hardware as shown in Figure 2. That is, the video processing device is assumed to be implemented by a procedure group candidate acquisition circuit 21, an action interval detection circuit 22, a procedure group candidate selection circuit 23, and a presentation processing circuit 24. Each of the procedure group candidate acquisition circuit 21, the action interval detection circuit 22, the procedure group candidate selection circuit 23, and the presentation processing circuit 24 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0020] The components of a video processing device are not limited to those implemented by dedicated hardware; the video processing device may also be implemented by software, firmware, or a combination of software and firmware. The software or firmware is stored as a program in the computer's memory. A computer refers to the hardware that executes programs, and includes, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor).

[0021] Figure 3 is a hardware configuration diagram of a computer when the video processing device is implemented by software or firmware. When the video processing device is implemented by software or firmware, programs that cause the computer to execute each of the processing procedures in the procedure group candidate acquisition unit 1, the action interval detection unit 2, the procedure group candidate selection unit 3, and the presentation processing unit 4 are stored in memory 31. The computer's processor 32 then executes the programs stored in memory 31.

[0022] Furthermore, Figure 2 shows an example in which each component of the video processing device is implemented by dedicated hardware, and Figure 3 shows an example in which the video processing device is implemented by software or firmware, etc. However, this is only one example, and some components of the video processing device may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0023] Next, the operation of the video processing device shown in Figure 1 will be explained. Figure 4 is a flowchart of the video processing method, which is the processing procedure of the video processing device. The procedure group candidate acquisition unit 1 acquires multiple procedure group candidates that are different from each other (step ST1 in Figure 4). Specifically, the procedure group candidate acquisition unit 1 acquires a request from an external source to create a series of procedures for executing tasks shown in a video. The procedure group candidate acquisition unit 1 then provides the creation request to the generating AI (Artificial Intelligence) and acquires multiple procedure group candidates from the generating AI. The procedure group candidate acquisition unit 1 outputs information indicating the multiple procedure group candidates to the procedure group candidate selection unit 3. The creator of the request to create a series of procedures may be, for example, the creator of the video, a third party other than the creator of the video, or a user who wants to check the content of the tasks shown in the video. Furthermore, if the creation requests corresponding to each of the multiple tasks are stored in, for example, a table (not shown), the procedure group candidate acquisition unit 1 may, instead of acquiring the creation requests from an external source, acquire the identification information of the tasks shown in the video and read the creation requests corresponding to the tasks indicated by the identification information from the table.

[0024] A task, for example, refers to work or an assignment, and the video shows the work or assignment being carried out. If the purpose of the task shown in the video is, for example, "to make coffee," then a request to create a series of steps might be, "Please provide three possible steps for making coffee."

[0025] When the generating AI receives a request from the procedure group candidate acquisition unit 1 to create a series of procedures, it creates multiple procedure group candidates corresponding to the creation request and outputs the multiple procedure group candidates to the procedure group candidate acquisition unit 1. For example, if the generating AI receives a request to create a series of procedures, such as "Please give me three candidate procedures for making coffee," it will create three procedure group candidates (1) to (3) as shown below.

[0026] Procedure group candidate (1): Pick up a cup → Pour coffee → Pour milk → Stir the coffee Procedure group candidate (2): Pick up a cup → Pour coffee → Pour water → Stir the coffee Procedure group candidate (3): Pick up a cup → Pour coffee → Pour milk → Pour sugar → Stir the coffee

[0027] Figure 5 is an explanatory diagram showing an example of three procedure group candidates created by the generation AI. In Figure 5, 〇 indicates a procedure. In the figure, the topmost procedure group candidate corresponds to procedure group candidate (1) including four procedures. In the figure, the second procedure group candidate from the top corresponds to procedure group candidate (2) including four procedures. In the figure, the third procedure group candidate from the top corresponds to procedure group candidate (3) including five procedures.

[0028] Each of the three procedure group candidates (1) to (3) includes the procedure names of the respective procedures it contains. For example, in the case of procedure group candidate (1), examples of the procedure name for the procedure of "Pick up a cup" include, for example, the procedure name of "Cup removal step". Examples of the procedure name for the procedure of "Pour coffee" include, for example, the procedure name of "Coffee pouring step". Examples of the procedure name for the procedure of "Pour milk" include, for example, the procedure name of "Milk pouring step". Examples of the procedure name for the procedure of "Stir the coffee" include, for example, the procedure name of "Stirring step".

[0029] Here, an example is shown where each of the procedure group candidates (1) to (3) includes information indicating a procedure name separately from a plurality of procedures. However, this is only an example, and each of the procedures that procedure group candidates (1) to (3) have may also serve as information indicating a procedure name. In the video processing apparatus shown in FIG. 1, the procedure group candidate acquisition unit 1 gives a creation request for a series of procedures for executing a task to the generation AI and acquires a plurality of procedure group candidates from the generation AI. However, this is only an example, and the procedure group candidate acquisition unit 1 may be given a plurality of procedure group candidates from outside the video processing apparatus, for example, via a network.

[0030] The action section detection unit 2 acquires a video in which the task is shown. The action section detection unit 2 detects an action section, which is an interval in the video where each of a series of procedures for executing the task may be shown, based on the change over time of the feature amount of the task shown in the video (step ST2 in FIG. 4). The action section detection unit 2 outputs the detection result of the action section to the procedure group candidate selection unit 3. As a method for detecting the action section by the action section detection unit 2, for example, a known technique called TAS (Temporal Action Segmentation) can be used. Hereinafter, the detection process of the action section by the action section detection unit 2 will be specifically described.

[0031] The feature amount extraction unit 2a acquires a video in which the task is shown. The feature amount extraction unit 2a extracts the feature amount of the task shown in the video. Since the extraction process of the feature amount itself is a known technique, detailed description thereof will be omitted. The feature amount extraction unit 2a outputs time-series data indicating the temporal change of the feature amount to the section group candidate acquisition unit 2b.

[0032] The section group candidate acquisition unit 2b acquires the time-series data from the feature amount extraction unit 2a. The section group candidate acquisition unit 2b acquires a plurality of different section group candidates based on the temporal change of the feature amount indicated by the time-series data, for example, using a technique called TAS. The section group candidate acquisition unit 2b outputs information indicating the plurality of section group candidates to the procedure group candidate selection unit 3 as the detection result of the action section.

[0033] In the video processing apparatus shown in FIG. 1, an example is shown in which the feature amount extraction unit 2a extracts the feature amount of the task, and the section group candidate acquisition unit 2b acquires a plurality of section group candidates based on the temporal change of the feature amount indicated by the time-series data. However, this is only an example, and the section group candidate acquisition unit 2b may give the time-series data output from the feature amount extraction unit 2a to a learned model that has learned the correspondence between the feature amount of the task and the action section, and acquire a plurality of section group candidates from the learned model. Examples of the learned model include neural models such as existing TAS models.

[0034] FIG. 6 is an explanatory diagram showing an example in which a plurality of interval group candidates are output from a neural model such as a TAS model. In FIG. 6, four types of interval group candidates IGC １ to IGC ４ are illustrated. Specifically, an interval group candidate IGC １ including action intervals (1), action intervals (3), and action intervals (5) to (7), an interval group candidate IGC ２ including action intervals (1), action intervals (4), and action intervals (5) to (7), an interval group candidate IGC ３ including action intervals (2), action intervals (3), and action intervals (5) to (7), and an interval group candidate IGC ４ including action intervals (2), action intervals (4), and action intervals (5) to (7) are illustrated.

[0035] The procedure group candidate selection unit 3 acquires information indicating a plurality of procedure group candidates from the procedure group candidate acquisition unit 1, and acquires the detection result of the action interval from the action interval detection unit 2. The procedure group candidate selection unit 3 selects a procedure group candidate corresponding to the action interval detected by the action interval detection unit 2 from among the plurality of procedure group candidates acquired by the procedure group candidate acquisition unit 1 (step ST3 in FIG. 4). The procedure group candidate selection unit 3 outputs information indicating the selected procedure group candidate to the presentation processing unit 4. Hereinafter, the selection process of the procedure group candidate by the procedure group candidate selection unit 3 will be specifically described.

[0036] The collation processing unit 3a acquires, as information indicating a plurality of procedure group candidates, information indicating, for example, procedure group candidates (1) to (3) from the procedure group candidate acquisition unit 1. The collation processing unit 3a acquires, as information indicating a plurality of interval group candidates, information indicating, for example, four types of interval group candidates IGC １ to IGC ４ from the interval group candidate acquisition unit 2b. The collation processing unit 3a converts the information indicating the procedure group candidates (1) to (3) into information in a certain feature space, and converts the information indicating the interval group candidates IGC １ to IGC ４ into information in the same feature space as the above-mentioned feature space. Since the process of converting these information into information in the feature space is a known technique, detailed description thereof will be omitted.

[0037] The matching processing unit 3a compares each of the procedure group candidates (1) to (3) and the interval group candidate IGC in the feature space described above. １ ~ IGC ４ Each of these is compared. Specifically, the comparison processing unit 3a compares each of the procedure group candidates (1) to (3) with the interval group candidate IGC in the above feature space. １ ~ IGC ４ The similarity between each of the following is calculated. The process of calculating the similarity between the procedure group candidate and the interval group candidate is a well-known technique, so a detailed explanation is omitted. The matching processing unit 3a outputs the similarity calculation result as the matching result to the procedure group candidate selection processing unit 3b.

[0038] The procedure group candidate selection processing unit 3b acquires information from the procedure group candidate acquisition unit 1 indicating multiple procedure group candidates, for example, information indicating procedure group candidates (1) to (3). The procedure group candidate selection processing unit 3b acquires information from the interval group candidate acquisition unit 2b indicating multiple interval group candidates, for example, four types of interval group candidate IGC １ ~ IGC ４ Information indicating the following is obtained. The procedure group candidate selection processing unit 3b obtains the similarity calculation result as the comparison result from the comparison processing unit 3a. Based on the comparison result from the comparison processing unit 3a, the procedure group candidate selection processing unit 3b selects procedure group candidates (1) to (3) and interval group candidate IGC. １ ~ IGC ４ From among these, the combination of procedure group candidate and interval group candidate with the highest similarity is selected. The procedure group candidate selection processing unit 3b selects the combination of procedure group candidate and interval group candidate with the highest similarity, for example, procedure group candidate (3) and interval group candidate IGC １ In combination with this, there are four possible interval group candidates IGC １ ~ IGC ４ From among them, the candidate interval group IGC １ Select this option.

[0039] Figure 7 is an action send date graph showing an example of interval group candidate selection by the procedure group candidate selection processing unit 3b. In Figure 7, interval group candidate IGC １This shows an example of what is selected. In Figure 7, the horizontal axis represents time, and the vertical axis represents the posterior probability. The procedure group candidate selection processing unit 3b selects the interval group candidate AGC. １ The information indicating the selected procedure group candidate (3) and the information indicating the selected procedure group candidate (3) are output to the presentation processing unit 4.

[0040] In the image processing device shown in Figure 1, the procedure group candidate selection processing unit 3b selects the combination of procedure group candidate and interval group candidate that has the highest similarity. The combination of procedure group candidate and interval group candidate is not limited to the combination with the highest similarity; the procedure group candidate selection processing unit 3b may, within a range that does not pose a practical problem, select, for example, the second-highest similar combination or the third-highest similar combination.

[0041] The presentation processing unit 4 obtains information indicating the selected interval group candidate and information indicating the selected procedure group candidate from the procedure group candidate selection processing unit 3b. For example, the presentation processing unit 4 obtains information indicating the interval group candidate IGC. １ Information indicating the and information indicating the procedure group candidate (3) are acquired. The presentation processing unit 4 displays the action interval included in the interval group candidate indicated by the acquired information on a display device (not shown) (step ST4 in Figure 4). Specifically, the presentation processing unit 4 determines that the interval group candidate indicated by the acquired information is the interval group candidate IGC １ If so, as shown in Figure 8, the candidate interval group IGC １ The action sections included in the information are, for example, action section (1), action section (3), and action sections (5) to (7), each displayed on a display device (not shown). The presentation processing unit 4 also displays the names of the procedures included in the procedure group candidate indicated by the acquired information on a display device (not shown) (step ST4 in Figure 4). Specifically, if the procedure group candidate indicated by the acquired information is procedure group candidate (3), the presentation processing unit 4 displays the procedure names "cup removal step", "coffee pouring step", "milk pouring step", "sugar pouring step", and "stirring step" on a display device (not shown), each as shown in Figure 8.

[0042] Figure 8 is an explanatory diagram showing an example of an action section and procedure name presented by the presentation processing unit 4. Users who view the display content of the display device can easily recognize the procedure name of the task shown in the video, thereby easily understanding the content of the task shown in the video. Therefore, users who view the display content of the display device can easily perform the task shown in the video.

[0043] In the above embodiment 1, the video processing device is configured to include a procedure group candidate acquisition unit 1 that acquires multiple procedure group candidates, each containing a possible sequence of steps for executing a task shown in a video and the name of each step in those steps; and an action section detection unit 2 that detects action sections in a video, which are sections in the video in which each of the sequence of steps for executing a task may be shown, based on the change in the feature quantities of the task shown in the video over time. The video processing device also includes a procedure group candidate selection unit 3 that selects a procedure group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit 1 that corresponds to an action section detected by the action section detection unit 2. Therefore, the video processing device can acquire the name of the step corresponding to the action section.

[0044] In Embodiment 1, the video processing device is configured such that the action interval detection unit 2 includes a feature extraction unit 2a that extracts feature quantities of a task shown in the video and outputs time-series data showing the temporal changes in these feature quantities, and an interval group candidate acquisition unit 2b that acquires multiple interval group candidates that include action intervals in which each of the steps for executing the task may be shown, based on the temporal changes in feature quantities shown by the time-series data output from the feature extraction unit 2a. Furthermore, the video processing device is configured such that the procedure group candidate selection unit 3 includes a matching processing unit 3a that matches each procedure group candidate acquired by the procedure group candidate acquisition unit 1 with each interval group candidate acquired by the interval group candidate acquisition unit 2b, and a procedure group candidate selection processing unit 3b that, based on the matching results of the matching processing unit 3a, selects an interval group candidate that includes an action interval in which each of the steps for executing the task may be shown from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit 1, and also selects a procedure group candidate that corresponds to the action interval included in the selected interval group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit 1. Therefore, the video processing device can select a group of candidate intervals containing action intervals in which each of the steps for performing a task may be shown, and a group of candidate steps corresponding to action intervals in which each of the steps may be shown.

[0045] In Embodiment 1, the video processing device is configured such that the action interval detection unit 2 includes a feature extraction unit 2a that extracts feature quantities of tasks shown in the video and outputs time-series data showing the temporal changes in these feature quantities, and an interval group candidate acquisition unit 2b that provides the time-series data output from the feature extraction unit 2a to a trained model that has learned the correspondence between task feature quantities and action intervals, and acquires a plurality of interval group candidates from the trained model that include action intervals in which each of the steps for executing a task may be shown. Furthermore, the video processing device is configured such that the procedure group candidate selection unit 3 includes a comparison processing unit 3a that compares each procedure group candidate acquired by the procedure group candidate acquisition unit 1 with each section group candidate acquired by the section group candidate acquisition unit 2b, and a procedure group candidate selection processing unit 3b that, based on the comparison result of the comparison processing unit 3a, selects a section group candidate from among a plurality of section group candidates acquired by the section group candidate acquisition unit 2b that includes an action section in which each of the steps in a series of procedures for executing a task is shown, and also selects a procedure group candidate from among a plurality of procedure group candidates acquired by the procedure group candidate acquisition unit 1 that corresponds to an action section included in the selected section group candidate. Therefore, the video processing device can select a section group candidate that includes an action section in which each of the steps in a series of procedures for executing a task may be shown, and a procedure group candidate that corresponds to an action section in which each of the steps in a series of procedures may be shown.

[0046] In Embodiment 1, the video processing device is configured such that the matching processing unit 3a calculates the similarity between each procedure group candidate acquired by the procedure group candidate acquisition unit 1 and each interval group candidate acquired by the interval group candidate acquisition unit 2b, and outputs the similarity calculation result as the matching result to the procedure group candidate selection processing unit 3b. Therefore, the video processing device can select a combination of procedure group candidates and interval group candidates with a high similarity.

[0047] In Embodiment 1, the video processing device is configured to include a presentation processing unit 4 that presents the action intervals included in the candidate interval group selected by the candidate interval group processing unit 3b, and the procedure names of the procedures included in the candidate interval group selected by the candidate interval group processing unit 3b. Therefore, the video processing device can present the action intervals and procedure names.

[0048] In Embodiment 1, the video processing device is configured such that the procedure group candidate acquisition unit 1 provides the generating AI with a request to create a series of procedures for executing a task shown in the video, and acquires multiple procedure group candidates from the generating AI. Therefore, the video processing device can acquire multiple procedure group candidates simply by providing the generating AI with a request to create a series of procedures.

[0049] In the image processing device shown in Figure 1, the matching processing unit 3a converts information indicating procedure group candidates (1) to (3) into information in a certain feature space, and interval group candidate IGC １ ~ IGC ４ The information indicating the above is converted into information in the same feature space as the feature space mentioned above. However, this is just one example, and the matching processing unit 3a may, for example, not convert the information indicating the procedure group candidates (1) to (3) and instead perform the interval group candidate IGC. １ ~ IGC ４ The information indicating the above may be transformed into a feature space of information indicating the candidate procedure group (1) to (3).

[0050] Embodiment 2. Embodiment 2 describes a video processing device in which the procedure group candidate acquisition unit 5 acquires a plurality of procedure group candidates based on text indicating the purpose of the task shown in the video.

[0051] Figure 9 is a configuration diagram showing an image processing device according to Embodiment 2. In Figure 9, the same reference numerals as in Figure 1 indicate the same or corresponding parts, so a detailed explanation is omitted. Figure 10 is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 2. In Figure 10, the same reference numerals as in Figure 2 indicate the same or corresponding parts, so a detailed explanation is omitted. The image processing device shown in Figure 9 includes a procedure group candidate acquisition unit 5, an action interval detection unit 2, a procedure group candidate selection unit 3, and a presentation processing unit 4.

[0052] The procedure group candidate acquisition unit 5 is implemented, for example, by the procedure group candidate acquisition circuit 25 shown in Figure 10. The procedure group candidate acquisition unit 5 acquires multiple procedure group candidates based on the text indicating the purpose of the task shown in the video and the action labels. The action labels are labels that indicate the action section, which is the section in which each of the series of steps for executing the task is shown, and the name of the step corresponding to that action section. The procedure group candidate acquisition unit 5 outputs information indicating the multiple procedure group candidates to the procedure group candidate selection unit 3.

[0053] In Figure 9, it is assumed that the components of the video processing device—the procedure group candidate acquisition unit 5, the action interval detection unit 2, the procedure group candidate selection unit 3, and the presentation processing unit 4—are each implemented by dedicated hardware as shown in Figure 10. That is, it is assumed that the video processing device is implemented by a procedure group candidate acquisition circuit 25, an action interval detection circuit 22, a procedure group candidate selection circuit 23, and a presentation processing circuit 24. Each of the procedure group candidate acquisition circuit 25, the action interval detection circuit 22, the procedure group candidate selection circuit 23, and the presentation processing circuit 24 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0054] The components of the video processing device are not limited to those implemented by dedicated hardware; the video processing device may also be implemented by software, firmware, or a combination of software and firmware. When the video processing device is implemented by software or firmware, a program that causes the computer to execute each of the processing procedures in the procedure group candidate acquisition unit 5, the action interval detection unit 2, the procedure group candidate selection unit 3, and the presentation processing unit 4 is stored in the memory 31 shown in Figure 3. Then, the processor 32 shown in Figure 3 executes the program stored in the memory 31.

[0055] Furthermore, Figure 10 shows an example in which each component of the video processing device is implemented by dedicated hardware, and Figure 3 shows an example in which the video processing device is implemented by software or firmware, etc. However, this is only one example, and some components of the video processing device may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0056] Next, the operation of the video processing device shown in Figure 9 will be explained. However, since all parts except the procedure group candidate acquisition unit 5 are the same as those of the video processing device shown in Figure 1, only the operation of the procedure group candidate acquisition unit 5 will be explained here.

[0057] The procedure group candidate acquisition unit 5 acquires text indicating the purpose of the task shown in the video, and action labels. If the purpose of the task shown in the video is, for example, "to make coffee," then the text indicating the purpose of the task is the text indicating "to make coffee." Action labels indicate the action section in which each of the steps for executing the task is shown, and the name of the step corresponding to that action section. Based on the text indicating the purpose of the task and the action labels, the procedure group candidate acquisition unit 5 acquires multiple procedure group candidates.

[0058] Specifically, the procedure group candidate acquisition unit 5 provides a language model such as LLM (Large Language Models) with text indicating the task's objective and action labels shown in the video, and acquires multiple procedure group candidates from the language model. The procedure group candidates include multiple possible steps in a series of steps for executing the task shown in the video, and the names of each step in those steps. During training, the language model learns procedure group candidates that correspond to the task's objective. That is, during training, the language model learns the correspondence between the task's objective, the action segments in which each of the steps for executing the task is shown, and the names of the steps corresponding to those action segments. During inference, when the language model receives text indicating the task's objective and labels from the procedure group candidate acquisition unit 5, it outputs multiple procedure group candidates to the procedure group candidate acquisition unit 5. The procedure group candidate acquisition unit 5 outputs information indicating the multiple procedure group candidates to the procedure group candidate selection unit 3.

[0059] The video processing device shown in Figure 9 demonstrates that, during training, the language model learns the correspondence between the task's objective, the action segments in which each of the steps for executing the task is shown, and the step names of the corresponding steps. However, this is merely one example, and the language model may not learn the correspondence between the task's objective, the action segments, and the step names of the corresponding steps. In this case, during inference, when the language model receives text indicating the task's objective and action labels shown in the video from the step group candidate acquisition unit 5, it outputs multiple step group candidates to the step group candidate acquisition unit 5 based on its common sense knowledge.

[0060] In the above embodiment 2, the video processing device is configured such that the procedure group candidate acquisition unit 5 acquires multiple procedure group candidates based on text indicating the purpose of the task shown in the video, and labels indicating action sections, which are sections in which each of the series of steps for executing the task is shown, and the procedure names of the steps corresponding to those action sections. Therefore, the video processing device can acquire multiple procedure group candidates simply by providing the text and labels to, for example, a language model.

[0061] Embodiment 3. Embodiment 3 describes an image processing apparatus comprising a feature extraction unit 6, a procedure name acquisition unit 7, a procedure name modification unit 8, and a presentation processing unit 4.

[0062] Figure 11 is a configuration diagram showing an image processing device according to Embodiment 3. In Figure 11, the same reference numerals as in Figure 1 indicate the same or corresponding parts, so a detailed explanation is omitted. Figure 12 is a hardware configuration diagram showing the hardware of the image processing device according to Embodiment 3. In Figure 12, the same reference numerals as in Figure 2 indicate the same or corresponding parts, so a detailed explanation is omitted. The image processing device shown in Figure 11 includes a feature extraction unit 6, a procedure name acquisition unit 7, a procedure name modification unit 8, and a presentation processing unit 4.

[0063] The feature extraction unit 6 is implemented, for example, by the feature extraction circuit 26 shown in Figure 12. The feature extraction unit 6 acquires a video in which the task is shown. The feature extraction unit 6 extracts the features of the task shown in the video. The feature extraction unit 6 outputs time-series data showing the change in features over time to the procedure name acquisition unit 7.

[0064] The procedure name acquisition unit 7 is implemented, for example, by the procedure name acquisition circuit 27 shown in Figure 12. The procedure name acquisition unit 7 acquires time-series data showing the time change of features from the feature extraction unit 6. The procedure name acquisition unit 7 provides the time-series data output from the feature extraction unit 6 to a trained model 9 that has learned the correspondence between the features of a task and the procedure names of the steps for executing the task, and acquires the procedure names of the series of steps for executing the task shown in the video from the trained model 9. The procedure name acquisition unit 7 outputs information indicating the procedure names to the procedure name correction unit 8.

[0065] During training, the trained model 9, given time-series data showing the temporal changes in the features of tasks shown in a video, and action labels, learns the correspondence between the features of tasks shown in a video, action intervals, and procedure names. During inference, when the trained model 9 is given time-series data output from the feature extraction unit 6 by the procedure name acquisition unit 7, it outputs to the procedure name acquisition unit 7 the action intervals, which are the intervals in which each of the series of steps for executing the task from which features are extracted by the feature extraction unit 6 is shown, and the procedure names of the steps corresponding to the action intervals. In the video processing device shown in Figure 11, the trained model 9 is located outside the video processing device. However, this is just one example, and the trained model 9 may be located inside the video processing device.

[0066] The procedure name correction unit 8 is implemented, for example, by the procedure name correction circuit 28 shown in Figure 12. The procedure name correction unit 8 obtains the procedure name of each procedure from the procedure name acquisition unit 7. The procedure name correction unit 8 corrects the procedure name obtained by the procedure name acquisition unit 7 using action labels that indicate the procedure name of the procedure corresponding to the action interval, which is the interval in which each of the series of procedures for executing the task from which features are extracted by the feature extraction unit 6 is shown. The procedure name correction unit 8 outputs information indicating the corrected procedure name to the presentation processing unit 4.

[0067] In Figure 11, it is assumed that the feature extraction unit 6, procedure name acquisition unit 7, procedure name correction unit 8, and presentation processing unit 4, which are components of the video processing device, are each implemented by dedicated hardware as shown in Figure 12. That is, it is assumed that the video processing device is implemented by a feature extraction circuit 26, a procedure name acquisition circuit 27, a procedure name correction circuit 28, and a presentation processing circuit 24. Each of the feature extraction circuit 26, procedure name acquisition circuit 27, procedure name correction circuit 28, and presentation processing circuit 24 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0068] The components of the video processing device are not limited to those implemented by dedicated hardware; the video processing device may also be implemented by software, firmware, or a combination of software and firmware. When the video processing device is implemented by software or firmware, a program for causing the computer to execute the respective processing procedures in the feature extraction unit 6, the procedure name acquisition unit 7, the procedure name modification unit 8, and the presentation processing unit 4 is stored in the memory 31 shown in Figure 3. Then, the processor 32 shown in Figure 3 executes the program stored in the memory 31.

[0069] Furthermore, Figure 12 shows an example in which each component of the video processing device is implemented by dedicated hardware, and Figure 3 shows an example in which the video processing device is implemented by software or firmware, etc. However, this is only one example, and some components of the video processing device may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0070] Next, the operation of the video processing device shown in Figure 11 will be explained. Figure 13 is a flowchart showing the video processing method, which is the processing procedure of the video processing device. The feature extraction unit 6 acquires a video in which the task is shown. The feature extraction unit 6 extracts the features of the task shown in the video, similar to the feature extraction unit 2a shown in Figure 1 (step ST11 in Figure 13). The feature extraction unit 6 outputs time-series data showing the change in features over time to the procedure name acquisition unit 7.

[0071] The procedure name acquisition unit 7 acquires time-series data showing the time evolution of features from the feature extraction unit 6. The procedure name acquisition unit 7 provides the time-series data to the trained model 9 and acquires from the trained model 9 action intervals, which are intervals that represent each of the series of steps for executing the task from which features are extracted by the feature extraction unit 6, and the procedure names of the steps corresponding to those action intervals (step ST12 in Figure 13). The procedure name acquisition unit 7 outputs action interval information indicating the action intervals and procedure name information indicating the procedure names to the procedure name correction unit 8.

[0072] The procedure name modification unit 8 obtains action interval information and procedure name information from the procedure name acquisition unit 7. In addition to the action intervals that show each of the steps in a series of steps for executing a task, the procedure name modification unit 8 obtains action labels that indicate the procedure names of the steps corresponding to the action intervals. The procedure name modification unit 8 modifies the procedure names obtained by the procedure name acquisition unit 7 using the action labels (step ST13 in Figure 13). The procedure name modification process by the procedure name modification unit 8 will be described in detail below.

[0073] Specifically, the procedure name modification unit 8 compares multiple action intervals indicated by the action interval information (hereinafter referred to as "acquired action intervals") with multiple action intervals indicated by the action labels (hereinafter referred to as "labeled action intervals") to identify identical action intervals. The procedure name modification unit 8 compares the procedure name of the procedure corresponding to the acquired action interval (hereinafter referred to as "acquired procedure name") with the procedure name indicated by the action label corresponding to the labeled action interval that is the same action interval as the acquired action interval (hereinafter referred to as "labeled procedure name"). If the acquired procedure name and the labeled procedure name match, the procedure name modification unit 8 outputs the action interval information indicating the acquired action interval and the procedure name information indicating the acquired procedure name to the presentation processing unit 4 without modifying the acquired procedure name. If the acquired procedure name and the labeled procedure name do not match, the procedure name modification unit 8 modifies the acquired procedure name to the labeled procedure name and outputs the action interval information indicating the acquired action interval and the procedure name information indicating the modified procedure name to the presentation processing unit 4.

[0074] The presentation processing unit 4 obtains action interval information and procedure name information from the procedure name correction unit 8. The presentation processing unit 4 displays the action interval indicated by the action interval information and the procedure name indicated by the procedure name information on a display device (not shown) (step ST14 in Figure 13).

[0075] In the above embodiment 3, the video processing device is configured to include a feature extraction unit 6 that extracts features of tasks shown in a video and outputs time-series data showing the temporal changes in these features, and a procedure name acquisition unit 7 that provides the time-series data output from the feature extraction unit 6 to a trained model 9 and obtains the procedure names of a series of steps for executing the tasks shown in the video from the trained model 9. Therefore, the video processing device can obtain the procedure names of steps corresponding to action intervals.

[0076] In Embodiment 3, the video processing device is configured to include a procedure name correction unit 8 that corrects the procedure names obtained by the procedure name acquisition unit 7 using labels indicating the procedure names of the procedures corresponding to the action intervals, which are sections in which each of the series of steps for executing a task from which features are extracted by the feature extraction unit 6 is shown. Therefore, the video processing device can optimize the procedure names.

[0077] In the video processing device shown in Figure 11, the procedure name acquisition unit 7 provides time-series data to a trained model 9 and obtains action intervals and procedure names from the trained model 9. However, this is just one example, and the procedure name acquisition unit 7 may also acquire labels indicating action intervals and the procedure names of the procedures corresponding to those action intervals from supplementary materials such as work manuals, standard procedure documents, or recipes. Specifically, the procedure name acquisition unit 7 provides the supplementary materials to a first language model and obtains labels from the first language model. The procedure name acquisition unit 7 may also provide text indicating the purpose of the task shown in the video and labels to a second language model and obtain multiple candidate procedure groups from the second language model.

[0078] In the video processing device shown in Figure 11, if the acquired procedure name and the labeled procedure name do not match, the procedure name correction unit 8 corrects the acquired procedure name to the labeled procedure name. However, this is only one example, and the procedure name correction unit 8 may also provide the language model with the action section information and procedure name information acquired from the procedure name acquisition unit 7, and instruct the language model to check the procedure name indicated by the procedure name information based on the text indicating the purpose of the task and the action label shown in the video. In this case, the language model determines the procedure name of the procedure corresponding to the action section in which each of the series of procedures is shown, based on the text indicating the purpose of the task and the action label shown in the video. The language model then compares the determined procedure name with the procedure name acquired by the procedure name acquisition unit 7. If the requested procedure name is, for example, "take cup, pour coffee, pour water, stir coffee", and the procedure name obtained by the procedure name acquisition unit 7 is, for example, "take plate, pour coffee, pour water, stir coffee", the language model will determine that "take cup" and "take plate" are different. The language model will then correct "take plate" in the procedure name obtained by the procedure name acquisition unit 7 to "take cup", and output the corrected procedure name, "take cup, pour coffee, pour water, stir coffee", to the presentation processing unit 4.

[0079] Alternatively, the procedure name modification unit 8 may provide the action interval information and procedure name information output from the procedure name acquisition unit 7 to the language model and obtain the modified procedure name from the language model. In this case, the language model checks whether the flow of action intervals indicated by the action interval information is natural, and based on the result of that check, modifies the procedure name if the flow of procedure names indicated by the procedure name information is not reasonable.

[0080] In the video processing device shown in Figure 11, the procedure name acquisition unit 7 provides time-series data to a trained model 9 and acquires action intervals and procedure names from the trained model 9. The procedure name acquisition unit 7 may further provide videos corresponding to each action interval to a VLM (Vision and Language Model) and acquire text information describing the content of the videos corresponding to each action interval. The procedure name acquisition unit 7 can inform the user of the content of the videos corresponding to each action interval by displaying the text information on a display device (not shown) via a presentation processing unit 4.

[0081] Furthermore, this disclosure allows for free combination of each embodiment, modification of any component in each embodiment, or omission of any component in each embodiment.

[0082] This disclosure includes a procedure group candidate acquisition unit that acquires multiple procedure group candidates, each containing a possible set of steps for performing a task shown in a video and the name of each step in those steps; and an action interval detection unit that detects action intervals in a video, which are sections in the video in which each of the steps for performing a task may be shown, based on the change in the feature quantities of the task shown in the video over time. The video processing device also includes a procedure group candidate selection unit that selects a procedure group candidate corresponding to an action interval detected by the action interval detection unit from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit, and can acquire the name of the step corresponding to the action interval, making it suitable for video processing devices and video processing methods.

[0083] 1 Procedure group candidate acquisition unit, 2 Action interval detection unit, 2a Feature extraction unit, 2b Interval group candidate acquisition unit, 3 Procedure group candidate selection unit, 3a Matching processing unit, 3b Procedure group candidate selection processing unit, 4 Presentation processing unit, 5 Procedure group candidate acquisition unit, 6 Feature extraction unit, 7 Procedure name acquisition unit, 8 Procedure name modification unit, 9 Trained model, 21 Procedure group candidate acquisition circuit, 22 Action interval detection circuit, 23 Procedure group candidate selection circuit, 24 Presentation processing circuit, 25 Procedure group candidate acquisition circuit, 26 Feature extraction circuit, 27 Procedure name acquisition circuit, 28 Procedure name modification circuit, 31 Memory, 32 Processor.

Claims

1. A video processing device comprising: a procedure group candidate acquisition unit that acquires multiple procedure group candidates, each including multiple possible steps for performing a task shown in a video and the name of each step in the multiple steps; an action section detection unit that detects action sections in the video, which are sections in the video in which each of the steps for performing the task may be shown, based on the change over time of the feature quantities of the task shown in the video; and a procedure group candidate selection unit that selects a procedure group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit that corresponds to an action section detected by the action section detection unit.

2. The video processing apparatus according to claim 1, wherein the action interval detection unit comprises a feature extraction unit that extracts feature quantities of a task shown in the video and outputs time-series data showing the temporal changes of the feature quantities, and an interval group candidate acquisition unit that acquires a plurality of interval group candidates that include action intervals in which each of the steps for executing the task may be shown, based on the temporal changes of the feature quantities shown by the time-series data output from the feature extraction unit, and the procedure group candidate selection unit comprises a matching processing unit that matches each procedure group candidate acquired by the procedure group candidate acquisition unit with each interval group candidate acquired by the interval group candidate acquisition unit, and a procedure group candidate selection processing unit that, based on the matching result of the matching processing unit, selects an interval group candidate that includes an action interval in which each of the steps for executing the task may be shown from among the plurality of procedure group candidates acquired by the procedure group candidate acquisition unit, and selects a procedure group candidate that corresponds to an action interval included in the selected interval group candidate from among the plurality of procedure group candidates acquired by the procedure group candidate acquisition unit.

3. The video processing apparatus according to claim 1, wherein the action interval detection unit comprises: a feature extraction unit that extracts feature quantities of a task shown in the video and outputs time-series data showing the temporal changes of the feature quantities; and an interval group candidate acquisition unit that provides the time-series data output from the feature extraction unit to a trained model that has learned the correspondence between the feature quantities of a task and action intervals, and acquires a plurality of interval group candidates from the trained model that include action intervals in which each of the steps for executing the task may be shown; and the procedure group candidate selection unit comprises: a matching processing unit that matches each procedure group candidate acquired by the procedure group candidate acquisition unit with each interval group candidate acquired by the interval group candidate acquisition unit; and a procedure group candidate selection processing unit that, based on the matching results of the matching processing unit, selects an interval group candidate from among the plurality of interval group candidates acquired by the interval group candidate acquisition unit that includes an action interval in which each of the steps for executing the task is shown, and selects a procedure group candidate from among the plurality of procedure group candidates acquired by the procedure group candidate acquisition unit that corresponds to an action interval included in the selected interval group candidate.

4. The video processing apparatus according to claim 2 or 3, characterized in that the matching processing unit calculates the similarity between each procedure group candidate acquired by the procedure group candidate acquisition unit and each interval group candidate acquired by the interval group candidate acquisition unit, and outputs the calculation result of the similarity as the matching result to the procedure group candidate selection processing unit.

5. The video processing apparatus according to any one of claims 2 to 4, further comprising a presentation processing unit that presents an action interval included in the candidate interval group selected by the candidate interval group processing unit, and the procedure name of a procedure included in the candidate interval group selected by the candidate interval group processing unit.

6. The video processing apparatus according to any one of claims 1 to 5, characterized in that the procedure group candidate acquisition unit provides a request to the generating AI (Artificial Intelligence) to create a series of procedures for executing the tasks shown in the video, and acquires the plurality of procedure group candidates from the generating AI.

7. The video processing apparatus according to any one of claims 1 to 5, characterized in that the procedure group candidate acquisition unit acquires the plurality of procedure group candidates based on text indicating the purpose of the task shown in the video, and labels indicating action sections which are sections in which each of the series of steps for executing the task is shown, and the procedure name of the step corresponding to the action section.

8. A video processing device comprising: a feature extraction unit that extracts features of tasks shown in a video and outputs time-series data showing the temporal changes of the features; and a procedure name acquisition unit that provides the time-series data output from the feature extraction unit to a trained model and obtains the procedure names of a series of steps for executing the tasks shown in the video from the trained model.

9. The video processing apparatus according to claim 8, characterized in that, during training, when the trained model is given time-series data showing the temporal changes in the features of a task shown in a video and labels indicating the names of the steps corresponding to the action intervals which are segments in which each of the steps for performing the task is shown, it learns the correspondence between the features of the task and the names of the steps, and during inference, when the time-series data output from the feature extraction unit is given, it outputs the names of the steps for performing the task from which the features have been extracted by the feature extraction unit.

10. The video processing apparatus according to claim 8 or 9, further comprising a procedure name modification unit that modifies the procedure name obtained by the procedure name acquisition unit using a label indicating the procedure name of the procedure corresponding to the action section, which is the section in which each of the series of steps for executing the task from which features are extracted by the feature extraction unit is shown.

11. A video processing method comprising: a procedure group candidate acquisition unit acquiring multiple procedure group candidates, each including multiple possible steps for executing a task shown in a video and the name of each step in the multiple steps; an action interval detection unit detecting action intervals in the video, which are intervals in the video in which each of the steps for executing the task may be shown, based on the change in the feature quantities of the task shown in the video over time; and a procedure group candidate selection unit selecting a procedure group candidate from among the multiple procedure group candidates acquired by the procedure group candidate acquisition unit that corresponds to the action interval detected by the action interval detection unit.

12. A video processing method comprising: a feature extraction unit extracting features of tasks shown in a video and outputting time-series data showing the temporal changes of the features; and a procedure name acquisition unit providing the time-series data output from the feature extraction unit to a trained model and obtaining the procedure names of a series of steps for executing the tasks shown in the video from the trained model.

Images