Information processing system, information processing method, and program

Abstract

The present technology pertains to an information processing system, an information processing method, and a program that make it possible to accurately segment a moving image while reducing a load on a user and processing. A SAM2 prediction unit predicts, with SAM2, a segmentation mask in a propagation-destination image captured at a time different from that of an operation target image by a camera that captured the operation target image, on the basis of a segmentation mask in the operation target image. An AOT prediction unit predicts, with AOT, a segmentation mask in the propagation-destination image on the basis of the segmentation mask in the operation target image. A zoom unit extracts a part of the propagation-destination image on the basis of the prediction results by the SAM2 prediction unit and the AOT prediction unit. An improvement unit predicts, with SAM2, a segmentation mask in the propagation-destination image on the basis of the part of the propagation-destination image extracted by the zoom unit, and obtains a final prediction result. The present technology can be applied to, for example, an information processing system or the like.

Description

Information Processing System, Information Processing Method, Program

[0001] The present technology relates to an information processing system, an information processing method, and a program, and particularly to an information processing system, an information processing method, and a program that can accurately perform segmentation of a moving image while reducing the load on users and processing.

[0002] In many important applications used in AR (Augmented Reality), VR (Virtual Reality), robots, self-driving vehicles, video editing, etc., segmentation of a moving image is necessary.

[0003] In the segmentation of a moving image, the position of a target object (entity) is recognized for each frame, but the target object in the moving image may look very different due to various factors such as movement, deformation, occlusion, and changes in lighting. Therefore, segmentation of a moving image is more difficult than segmentation of a still image.

[0004] As a method for segmenting a moving image, for example, there is a method of extracting the silhouette of a subject from a moving image by the background difference method (BSM (Background Subtraction Method)) (see, for example, Patent Document 1). However, it is difficult to appropriately segment an object such as a knife whose light reflection amount varies by part by such a method.

[0005] As a method for segmenting a moving image, there is also a method of using a deep planning network that has been fine-tuned in advance by additional learning data or data integration. In this case, the processing time often becomes long and the learning cost becomes high.

[0006] Another method for segmenting video is to use foundation models such as SAM (Segment Anything Model). SAM is a zero-shot learning foundation model and does not require training data. However, the user needs to input prompts such as bounding boxes, points, and masks. Therefore, with this method, the user has to input prompts for every frame of the video, which is cumbersome.

[0007] SAM2, the successor to SAM, can propagate a segmentation mask—a region containing a specific object within an image determined by user prompts for that frame—between frames of a video. However, to achieve accurate segmentation, the user must manually and iteratively refine SAM2 by correcting inaccurate segmentation mask predictions.

[0008] Japanese Patent Publication No. 2022-099127

[0009] Therefore, there is a demand for a method that accurately segments moving images while reducing the burden on users and processing power, but such demands have not yet been adequately met.

[0010] This technology was developed in light of these circumstances, and aims to enable accurate segmentation of moving images while reducing the burden on users and processing.

[0011] An information processing system representing one aspect of this technology comprises: a first prediction unit that predicts a predetermined region in a second image taken at a different time from the first image by the same imaging device that took the first image, based on a predetermined region in a first image, using a first method; a second prediction unit that predicts a predetermined region in a second image using a second method, based on the predetermined region in the first image; an extraction unit that extracts a portion of the second image based on the prediction results from the first and second prediction units; and an improvement unit that predicts the predetermined region in the second image using a third method and obtains a final prediction result based on the portion of the second image extracted by the extraction unit.

[0012] One aspect of this technology is an information processing method in which an information processing system predicts a predetermined region in a second image taken at a different time from the first image by a camera that took the first image, based on a predetermined region in the first image, using a first method and a second method, respectively; extracts a part of the prediction target based on the prediction results of the first method and the prediction results of the second method; and predicts the predetermined region in the second image using a third method based on the part of the prediction target, thereby obtaining a final prediction result.

[0013] One aspect of this technology is a program that causes a computer to perform a process that includes predicting a predetermined region in a second image, taken at a different time from the first image by the same camera that took the first image, based on a predetermined region in the first image, using a first method and a second method, respectively; extracting a portion of the prediction target based on the prediction results of the first method and the second method; and predicting the predetermined region in the second image using a third method based on the portion of the prediction target, thereby obtaining a final prediction result.

[0014] In one aspect of this technology, based on a predetermined region in a first image, the predetermined region in a second image, which is captured at a different time from the first image by the same camera that captured the first image, is predicted by a first method and a second method, respectively. Based on the prediction results of the first method and the second method, a portion of the second image is extracted, and based on the portion of the second image, the predetermined region in the second image is predicted by a third method, resulting in a final prediction result.

[0015] An information processing system may be a standalone device or a module incorporated into another device.

[0016] This figure shows an example of image segmentation by extracting the silhouette of a subject. This figure shows an example of error propagation. This figure shows an example of user correction of prediction results. This figure shows an example of camera arrangement for capturing moving images used in an information processing system to which this technology is applied. This block diagram shows an example configuration of one embodiment of an information processing system to which this technology is applied. This figure explains the overview of the segmentation process. This is a flowchart explaining the segmentation process. This figure explains prompt input operations for the target image. This is a flowchart explaining the details of the mask extraction process. This block diagram shows an example configuration of the time-direction propagation unit. This figure shows examples of SAM2 masks and AOT masks. This figure shows an example of a combined mask. This figure shows an example of the FoV region. This figure shows an example of an input image to SAM2. This figure shows an example of an improved mask. This figure explains the effect of the time-direction propagation unit. This figure shows an example of the results of the time-direction propagation process. This is a flowchart explaining the details of the time-direction propagation process. This figure shows an example of displaying a segmentation mask. This block diagram shows an example configuration of the spatial-direction propagation unit. This figure shows an example of selecting a reference image in the same-group spatial-direction propagation process. This figure shows an example of images of the same frame captured by two cameras. This figure shows an example of images from each group of cameras. This figure shows an example of the results of the same-group spatial-direction propagation process. This figure shows examples of SAM2 and AOT masks when the camera score is high. This figure shows an example of a silhouette. This figure shows an example of a zoomed image. This is a flowchart explaining spatial direction propagation processing. This is a block diagram showing an example of a computer hardware configuration.

[0017] The following describes the embodiments for implementing this technology. The explanation will be given in the following order: 1. Explanation of image segmentation by extracting the silhouette of the subject 2. Explanation of video segmentation using SAM2 3. One embodiment 4. Explanation of a computer to which this technology is applied

[0018] <1. Explanation of image segmentation by extracting the silhouette of the subject> Figure 1 shows an example of image segmentation by extracting the silhouette of the subject.

[0019] In the example shown in Figure 1, segmentation is performed on an image 11 in which a person 11b holding a sword 11a is the subject. In this case, a silhouette 12a is extracted from the image 11 using a method such as background subtraction, and a silhouette image 12 is generated as the segmented image in which the silhouette 12a is represented in white and the area 12b other than the silhouette 12a is represented in black. Because the amount of light reflected differs depending on the part of the sword 11a, as shown in Figure 1, an error occurs between the shape of the area 21 corresponding to the sword 11a in the silhouette 12a and the shape of the sword 11a.

[0020] As described above, with the method of extracting the silhouette of the subject, it is difficult to accurately segment objects such as the sword 11a, where the amount of light reflected differs depending on the part.

[0021] <2. Explanation of video segmentation using SAM2> <Example of error propagation> Figure 2 shows an example of error propagation in video segmentation using SAM2.

[0022] As mentioned above, SAM2 can propagate segmentation masks. Specifically, SAM2 can predict the segmentation mask of the target image using the segmentation masks of a predetermined number of frames preceding that image.

[0023] Therefore, as shown in Figure 2, once a segmentation mask 31a is set in the first frame's image 31, the segmentation mask for the next frame's image 32 is predicted based on that segmentation mask 31a.

[0024] However, as shown in Figure 2, if an error occurs during this prediction and the segmentation mask is not predicted, this error propagates to the prediction of images 33 and 34, which are frames after image 32, and the segmentation mask is not predicted. Therefore, the user needs to correct the prediction result of the segmentation mask where the error occurred.

[0025] <Example of correcting prediction results> Figure 3 shows an example of user-initiated correction of prediction results in video segmentation using SAM2.

[0026] In the example shown in Figure 3, segmentation of a moving image composed of four frames, 51 to 54, is performed.

[0027] In this case, the user first specifies, for example, point 51a of the dog's tongue in image 51 as a point to be included in the segmentation mask, as shown in Figure 3A, and specifies the three points 51b to 51d other than the dog's tongue as points not to be included in the segmentation mask. As a result, the region 61 of the dog's tongue, including point 51a in image 51, is extracted as the segmentation mask. Next, based on region 61, the region 62 corresponding to point 51a in image 52 of the next frame of image 51 is predicted as the segmentation mask, as shown in Figure 3A.

[0028] Furthermore, as shown in Figure 3A, if an error occurs during the prediction of image 53, the next frame after image 52 based on region 62, and the segmentation mask is not predicted, then the segmentation mask will also not be predicted for image 54, the next frame after image 53.

[0029] Therefore, in this case, as shown in Figure 3B, the user specifies point 53a on the dog's tongue in image 53 as a point to be included in the segmentation mask. As a result, the predicted segmentation mask for image 53 is modified to include the region 63 of the dog's tongue that contains point 53a in image 53. Consequently, based on region 63, as shown in Figure 3B, region 64 in image 54, the next frame after image 53, which corresponds to region 63, is predicted as the segmentation mask.

[0030] <3. One Embodiment> <Example of Camera Arrangement> Figure 4 shows an example of camera arrangement for capturing moving images used in an information processing system to which this technology is applied.

[0031] As shown in Figure 4, the moving image is acquired, for example, by capturing the subject 65 from multiple directions within a 360-degree range in the up, down, left, and right directions in a volumetric capture studio 70. In this case, the moving image is a 4D asset composed of multiple images in which at least one position in the 3D spatial direction and the temporal direction is different.

[0032] In the example shown in Figure 4, 31 cameras 71-0 to 71-30 (imaging devices) are arranged in the volumetric capture studio 70. Each of these cameras 71-0 to 71-30 captures 31 images 81-0 to 81-30 for each frame. As a result, a moving image consisting of N frames (where N is multiple) of images 81-0 to 81-30 is captured.

[0033] In Figure 4, frame #i (where i is a non-negative integer) represents the i-th frame from the beginning. This is also true in Figure 6, which will be described later. In the following, when there is no need to distinguish between cameras 71-0 to 71-30, they will be collectively referred to as camera 71. When there is no need to distinguish between images 81-0 to 81-30, they will be collectively referred to as image 81. In the following, an example will be described in which there are 31 cameras 71 placed in the volumetric capture studio 70, but the number of cameras 71 can be any number of 2 or more.

[0034] <Example of Information Processing System Configuration> Figure 5 is a block diagram showing an example configuration of an information processing system to which this technology is applied.

[0035] The information processing system 100 in Figure 5 comprises an operation unit 101, a segmentation unit 102, a time-direction propagation unit 103, a memory bank 104, a spatial-direction propagation unit 105, and a display unit 106. The information processing system 100 performs segmentation processing to segment the moving images captured by the 31 cameras 71 in Figure 4.

[0036] Specifically, each part of the information processing system 100 is configured to acquire, as needed, a moving image consisting of N-frame images 81 captured by each of the 31 cameras 71.

[0037] The operation unit 101 accepts user input. For example, the operation unit 101 accepts user input of prompts for the target image, which is an image of a predetermined frame from a predetermined camera 71, displayed on the display unit 106. The operation unit 101 then supplies the prompts input by the user to the segmentation unit 102.

[0038] The operation unit 101 accepts an operation by the user to input whether the segmentation mask of the target image displayed on the display unit 106 is correct or incorrect. The operation unit 101 supplies correct / incorrect information to the segmentation unit 102. The operation unit 101 also accepts an operation by the user to input whether the segmentation mask of an image 81 other than the target image displayed on the display unit 106 is correct or incorrect. The operation unit 101 updates the memory bank 104 based on the correct / incorrect information.

[0039] The segmentation unit 102 (region extraction unit) performs a mask extraction process to extract the segmentation mask of the target image using SAM2 based on prompts supplied from the operation unit 101. Specifically, the segmentation unit 102 extracts the segmentation mask of the target image using SAM2 based on prompts. The segmentation unit 102 supplies mask region information indicating the position and shape of the extracted segmentation mask on the image 81 to the display unit 106. Based on the correct / incorrect information supplied from the operation unit 101, the segmentation unit 102 supplies the mask information of the segmentation mask of the target image to the memory bank 104 for storage. The mask information of the segmentation mask is information indicating the characteristics and context of the objects included in that segmentation mask.

[0040] The time-direction propagation unit 103 performs time-direction propagation processing (multi-frame propagation processing) to propagate the segmentation mask of the target image in the time direction using a dual-model self-reinforced memory algorithm. Specifically, the time-direction propagation unit 103 propagates the segmentation mask of the target image in the time direction using SAM2 and AOT (Associating Object with Transformers) while referring to and updating the memory bank 104. Details of the time-direction propagation processing will be described later with reference to Figures 10 to 18. The time-direction propagation unit 103 supplies the mask region information of the segmentation mask propagated by the time-direction propagation processing to the display unit 106.

[0041] Memory bank 104 consists of SAM2 memory bank 104a and AOT memory bank 104b. SAM2 memory bank 104a and AOT memory bank 104b store the mask information and prompts for the segmentation mask of each image 81.

[0042] The SAM2 memory bank 104a is referenced and updated when the time propagation unit 103 and the spatial propagation unit 105 propagate the segmentation mask using SAM2. The AOT memory bank 104b is referenced and updated when the time propagation unit 103 and the spatial propagation unit 105 propagate the segmentation mask using AOT.

[0043] The spatial propagation unit 105 (spatial prediction unit) performs spatial propagation processing (cross-camera propagation processing) to propagate the segmentation mask propagated in the time direction by the temporal propagation unit 103 in the spatial direction. Specifically, the spatial propagation unit 105 propagates the segmentation mask propagated in the time direction in the spatial direction using SAM2 and AOT while referring to and updating the memory bank 104. Details of the spatial propagation processing will be described later with reference to Figures 19 to 28. The spatial propagation unit 105 supplies the mask region information of the segmentation mask propagated by the spatial propagation processing to the display unit 106.

[0044] The display unit 106 displays the operation target image. The display unit 106 also displays a segmentation mask based on the mask area information supplied from the segmentation unit 102, the time-direction propagation unit 103, or the space-direction propagation unit 105. Specifically, the display unit 106 highlights the segmentation mask of the corresponding image 81 based on the mask area information.

[0045] As described above, in the information processing system 100, the segmentation unit 102 performs mask extraction processing based on the prompt input by the user. Therefore, the segmentation unit 102 can be realized by a CPU (Central Processing Unit) instead of an expensive GPU (Graphics Processing Unit). Thus, by realizing the segmentation unit 102 by the CPU and realizing the time-direction propagation unit 103 and the space-direction propagation unit 105 by the GPU, the resources of the GPU can be concentrated on the time-direction propagation processing and the space-direction propagation processing. That is, the GPU efficiency can be improved.

[0046] <Overview of Segmentation Processing> FIG. 6 is a diagram for explaining the overview of the segmentation processing by the information processing system 100 of FIG. 5.

[0047] In the example of FIG. 6, the operation target image is the image 81-0 of frame #0 captured by the camera 71-0.

[0048] In this case, in the segmentation processing, first, the segmentation unit 102 extracts the segmentation mask of the operation target image. Next, the time-direction propagation unit 103 propagates the segmentation mask in the time direction T. That is, the segmentation masks of the images of frame #0 to frame #N of the image 81-0 are predicted. Then, the space-direction propagation unit 105 propagates the segmentation mask in the space direction S for each frame. As a result, the segmentation of each of the N frames constituting the moving image captured by each of the 31 cameras 71 is completed.

[0049] <Explanation of Segmentation Process> Figure 7 is a flowchart illustrating the segmentation process performed by the information processing system 100 in Figure 5.

[0050] In step S11 of Figure 7, the segmentation unit 102 performs a mask extraction process. Details of this mask extraction process will be explained later with reference to Figure 9. In step S12, the time propagation unit 103 performs a time propagation process based on the segmentation mask extracted by the mask extraction process in step S11. Details of this time propagation process will be explained later with reference to Figure 18.

[0051] In step S13, the time propagation unit 103 displays the segmentation mask by supplying the mask region information of the segmentation mask propagated by the time propagation process in step S12 to the display unit 106. The user performs an operation to input correctness or incorrectness for the segmentation mask displayed on the display unit 106. The operation unit 101 receives this operation and recognizes the correctness or incorrectness of each segment.

[0052] In step S14, the operation unit 101 updates the memory bank 104 based on whether each recognized segmentation mask is correct or incorrect. Specifically, the operation unit 101 adds an error flag to the mask information of the segmentation masks that are recognized as incorrect and stored in the SAM2 memory bank 104a and the AOT memory bank 104b.

[0053] In step S15, the spatial propagation unit 105 performs spatial propagation on the segmentation mask that was propagated in the time direction by the temporal propagation process in step S12. Then the segmentation process is completed.

[0054] <Explanation of prompt input operations for the target image> Figure 8 is a diagram illustrating the user's prompt input operations for the target image.

[0055] As shown in Figure 8, when the target image 120 is displayed on the display unit 106, the user prompts for input of points 121-1 to 121-3 to be included in the segmentation mask and points 122-1 and 122-2 not to be included in the segmentation mask. The operation unit 101 receives this input and supplies points 121-1 to 121-3 to be included in the segmentation mask and points 122-1 and 122-2 not to be included in the segmentation mask to the segmentation unit 102.

[0056] <Detailed explanation of the mask extraction process> Figure 9 is a flowchart that explains the details of the mask extraction process in step S11 of Figure 7.

[0057] In step S31 of Figure 9, the segmentation unit 102 determines whether or not a prompt has been input from the operation unit 101. If it is determined in step S31 that a prompt has been input, the process proceeds to step S32.

[0058] In step S32, the segmentation unit 102 performs annotation on the target image based on the prompt. In step S33, the segmentation unit 102 extracts a segmentation mask of the target image using SAM2, based on the target image that was annotated in step S32.

[0059] In step S34, the segmentation unit 102 displays the segmentation mask by supplying the mask region information of the segmentation mask extracted in step S33 to the display unit 106. The user performs an operation to input correct or incorrect while looking at this segmentation mask. The operation unit 101 receives this operation and supplies correct / incorrect information representing the correct or incorrect input by the user to the segmentation unit 102.

[0060] In step S35, the segmentation unit 102 determines whether the correct / incorrect information supplied from the operation unit 101 is correct. If it is determined in step S35 that the information is correct, the process proceeds to step S36.

[0061] In step S36, the segmentation unit 102 supplies the mask information of the segmentation mask extracted in step S33 to the memory bank 104, which stores it in the SAM2 memory bank 104a and the AOT memory bank 104b. The mask extraction process then ends, and the process returns to the process in step S11 in Figure 7, and proceeds to the process in step S12.

[0062] On the other hand, if it is determined in step S31 that no prompt has been entered yet, or if it is determined in step S35 that the correct / incorrect information does not indicate that the information is correct, the process returns to step S31 and the subsequent processes are repeated.

[0063] <Example of configuration of the time propagation unit> Figure 10 is a block diagram showing an example of the configuration of the time propagation unit 103.

[0064] The time-direction propagation unit 103 (time-direction prediction unit) in Figure 10 comprises a SAM2 prediction unit 141, an AOT prediction unit 142, a merging unit 143, a zoom unit 144, and an improvement unit 145.

[0065] The SAM2 prediction unit 141 uses the image to be operated on as a reference image (first image) and, while referring to the SAM2 memory bank 104a, propagates the segmentation mask of the reference image in both forward and backward directions in time using SAM2.

[0066] Specifically, the SAM2 prediction unit 141 (first prediction unit) sequentially selects images 81 from the SAM2 memory bank 104a that are at different frames (times) from the target image captured by the camera 71 that captured the target image, and determines them to be the destination image 81 (second image). Hereafter, the camera 71 that captured the target image will be referred to as the reference camera (reference imaging device).

[0067] The SAM2 prediction unit 141 reads mask information for images 81 from the SAM2 memory bank 104a, for frames taken by the reference camera that are either before or after the target image 81. Based on this mask information and the target image 81, the SAM2 prediction unit 141 uses SAM2 to predict the segmentation mask in the target image 81.

[0068] The SAM2 prediction unit 141 evaluates the prediction results by calculating object scores for the prediction results. For example, the predicted value of IoU (Intersection over Union) is used as the object score. The SAM2 prediction unit 141 retains only the segmentation masks corresponding to object scores that exceed a predetermined threshold, resets the other segmentation masks to empty, and supplies them to the merging unit 143 as SAM2 masks.

[0069] The AOT prediction unit 142 uses the target image as a reference image and, while referring to the AOT memory bank 104b, propagates the segmentation mask of the reference image in both forward and backward directions using AOT.

[0070] Specifically, the AOT prediction unit 142 (second prediction unit) determines the target image 81, similar to the SAM2 prediction unit 141. The AOT prediction unit 142 reads the mask information of the image 81 from the AOT memory bank 104b, which is captured by the reference camera and is either before or after the target image 81. Based on this mask information and the target image 81, the AOT prediction unit 142 predicts the segmentation mask in the target image 81 using AOT.

[0071] Furthermore, AOT has a more complex architecture than SAM2, employing LSTT (Long Short-Term Transformer) for hierarchical object matching and propagation. AOT is also trained on a different dataset than SAM2's SA-1B (Segment Anything 1-Billion mask dataset). Therefore, AOT can capture features of objects included in the segmentation mask in a different way than SAM2, and perform tracking in a different way than SAM2.

[0072] The AOT prediction unit 142 evaluates the prediction results by calculating object scores for the prediction results. For example, the object scores used are values ​​calculated from the output values ​​of the softmax function. The AOT prediction unit 142 retains only the segmentation masks corresponding to object scores that exceed a predetermined threshold, resets the other segmentation masks to empty, and supplies them to the merging unit 143 as AOT masks.

[0073] The thresholds in the SAM2 prediction unit 141 and the AOT prediction unit 142 are hyperparameters that control the trade-off between accuracy and reproducibility, and can be determined, for example, by the user. If the object scores of the segmentation masks of all N frames of images 81 captured by the same camera 71 do not exceed a predetermined threshold, the camera 71 is flagged. The user checks the images 81 captured by the flagged camera 71 and performs annotation by operating the operation unit 101.

[0074] The merging unit 143 merges the SAM2 mask supplied from the SAM2 prediction unit 141 and the AOT mask supplied from the AOT prediction unit 142, and supplies the mask region information of the resulting merged mask to the zoom unit 144.

[0075] The zoom unit 144 determines the smallest bounding box based on the mask area information supplied from the merging unit 143. The zoom unit 144 determines the FoV (Field-of-View) area that includes that bounding box. The zoom unit 144 zooms in on the destination image 81 by extracting the FoV area from the destination image 81.

[0076] The zoom unit 144 extracts multiple points to be used as prompts in SAM2 from the merged mask in the zoomed image 81 according to a predetermined algorithm. These points are extracted in such a way that the accuracy of SAM2's prediction of the segmentation mask for a single frame is improved. For example, 25% of the points to be extracted are extracted using k-medoids, 50% are extracted using the OpenCV corner detection algorithm, and 25% are extracted randomly. The final number of points to be extracted is determined based on the size of the objects included in the segmentation mask, for example, eight or more. The zoom unit 144 supplies the FoV region and the points as prompts to the improvement unit 145.

[0077] The improvement unit 145 repeats the prediction of the segmentation mask of the FoV region by SAM2 a predetermined number of times (for example, 5 times) based on the prompt. The first prediction in the repeated predictions is performed using the FoV region and prompt supplied from the zoom unit 144, and the second and subsequent predictions are performed using the FoV region and prompt extracted by the same process as the zoom unit 144 based on the previous prediction result.

[0078] The improvement unit 145 can correct errors in the predicted segmentation mask by repeating the prediction process. As a result, the accuracy of the segmentation mask predicted by the improvement unit 145 is improved compared to the SAM2 mask, AOT mask, and combined mask. The improvement unit 145 uses the improved mask, which is the segmentation mask obtained as a result of repeated predictions, as the final prediction result.

[0079] The improvement unit 145 supplies the mask region information of the improvement mask to the display unit 106 as the mask region information of the segmentation mask of the target image 81. The improvement unit 145 supplies the mask information of the improvement mask to the SAM2 memory bank 104a and the AOT memory bank 104b as the mask information of the segmentation mask of the target image 81 and stores them.

[0080] As described above, the time propagation unit 103 can improve the prediction accuracy of the segmentation mask without requiring any annotation modification operations by the user. In other words, the time propagation unit 103 can self-reinforce its segmentation mask prediction. Furthermore, the prediction results from the self-reinforced prediction are stored in the memory bank 104 and used for prediction in the next propagation destination image 81, so the prediction of the time propagation unit 103 continues to self-reinforce due to the so-called flywheel effect.

[0081] In contrast, with SAM2 prediction, as described above in Figures 2 and 3, the user needs to correct the prediction result for the segmentation mask where the error occurred. Correction instructions, which are additional prompts entered by the user, are stored in the memory bank, and the prediction is improved using the mask information corresponding to those correction instructions.

[0082] <Examples of SAM2 masks and AOT masks> Figure 11 shows examples of SAM2 masks and AOT masks.

[0083] In the example shown in Figure 11, the main object included in the propagating segmentation mask is the sword 160. In this case, the SAM2 prediction unit 141 predicts only the handle portion of the sword 160 as the SAM2 mask 161 of the propagation destination image 81, for example, as shown in A of Figure 11. In other words, the prediction by the SAM2 prediction unit 141 has failed.

[0084] On the other hand, AOT can capture different characteristics of the blade 160 than SAM2 and perform tracking differently from SAM2. Therefore, the AOT prediction unit 142 predicts the blade portion of the blade 160 as the AOT mask 162 of the propagation destination image 81, for example, as shown in Figure 11B. In other words, the prediction by the AOT prediction unit 142 is not perfect, but it is not a failure.

[0085] <Example of a combined mask> Figure 12 shows an example of a combined mask.

[0086] In the example shown in Figure 12, the SAM2 prediction unit 141 predicts the SAM2 mask 161 shown in Figure 11, and the AOT prediction unit 142 predicts the AOT mask 162. In this case, as shown in Figure 12, the merging unit 143 merges the SAM2 mask 161 and the AOT mask 162 to generate a merged mask 163.

[0087] <Example of FoV region> Figure 13 shows an example of an FoV region.

[0088] In the example shown in Figure 13, the merger unit 143 generates the merged mask 163 shown in Figure 12. In this case, the zoom unit 144 determines the smallest bounding box 181 that includes the merged mask 163. Then, the zoom unit 144 determines the FoV region 182 that includes the bounding box 181 at its center.

[0089] The size of one side of the FoV region 182 is set to, for example, 1040 pixels. If the size of the FoV region 182 is smaller than the size of the input image suitable for the SAM2 architecture, or if the mask size ratio in the improvement unit 145 is smaller than a predetermined ratio, the accuracy of the prediction in the improvement unit 145 will deteriorate. The mask size ratio is the ratio of the segmentation mask to the image. Therefore, the mask size ratio in the improvement unit 145 is the ratio of the size of the combined mask 163 to the size of the FoV region 182.

[0090] <Examples of input images to SAM2 in the SAM2 prediction unit and improvement unit> Figure 14 shows examples of input images to SAM2 in the SAM2 prediction unit 141 and the improvement unit 145.

[0091] In the example shown in Figure 14, the zoom unit 144 extracts the FoV region 182 from the propagation destination image 81 in Figure 13. In this case, as shown in Figure 14A, the input image to SAM2 in the SAM2 prediction unit 141 is the entire propagation destination image 81. On the other hand, as shown in Figure 14B, the input image to SAM2 in the improvement unit 145 is the FoV region 182 of the transmission destination image 81 that is suitable as input to SAM2. As a result, the accuracy of the improvement mask is higher than that of the SAM2 mask.

[0092] <Example of an improved mask> Figure 15 shows an example of an improved mask.

[0093] In the example shown in Figure 15, the input image to SAM2 in the improvement unit 145 is the FoV region 182. In this case, as shown in Figure 15, the improvement mask 190 of the propagation destination image 81 is approximately the same as the region of the blade 160.

[0094] The mask information of this improved mask 190 is stored in the SAM2 memory bank 104a and the AOT memory bank 104b, and is used for prediction by the SAM2 prediction unit 141 and the AOT prediction unit 142. Therefore, it is possible to prevent the propagation of errors as explained in Figure 2 and prevent the deterioration of predictions.

[0095] <Explanation of the effects of the time propagation unit> Figure 16 is a diagram illustrating the effects of the time propagation unit 103.

[0096] As shown in Figure 16, in the time propagation unit 103, prediction is performed by SAM2 in the SAM2 prediction unit 141 and by AOT in the AOT prediction unit 142, and the resulting SAM2 mask and AOT mask are combined. Therefore, the robustness of the segmentation mask prediction can be improved compared to the case where only SAM2 prediction is performed.

[0097] The improvement unit 145 improves the segmentation mask based on a combined mask formed by combining the SAM2 mask and the AOT mask. Therefore, the segmentation mask prediction can be self-reinforced.

[0098] The improved mask obtained as a result of the improvements is stored in the SAM2 memory bank 104a and the AOT memory bank 104b, and is used for prediction by SAM2 in the SAM2 prediction unit 141 and for prediction by AOT in the AOT prediction unit 142. Therefore, the segmentation mask prediction is continuously reinforced.

[0099] <Example of the result of time-direction propagation processing> Figure 17 shows an example of the result of time-direction propagation processing by the time-direction propagation unit 103.

[0100] In the example shown in Figure 17, the reference camera captures seven frames of image 81, including the target image 200. In this case, as shown in Figure 17, the time propagation unit 103 predicts the segmentation masks 202 and 203 of the two frames of image 81 preceding the target image 200, based on the segmentation mask 201 of the target image 200. The time propagation unit 103 also predicts the segmentation masks 204 to 207 of the four frames of image 81 following the target image 200, based on the segmentation mask 201.

[0101] <Detailed explanation of time propagation process> Figure 18 is a flowchart that explains the details of the time propagation process in step S12 of Figure 7.

[0102] In step S50 of Figure 18, the time propagation unit 103 determines the propagation direction to be either forward or backward in time, and determines the image 81 of the next frame of the image to be manipulated in that propagation direction as the destination image 81.

[0103] In step S51, the SAM2 prediction unit 141 generates a SAM2 mask for the propagation destination image 81 while referring to the SAM2 memory bank 104a and supplies it to the merging unit 143.

[0104] In step S52, the AOT prediction unit 142 generates an AOT mask for the propagation destination image 81 while referring to the AOT memory bank 104b and supplies it to the merging unit 143.

[0105] In step S53, the merging unit 143 combines the SAM2 mask generated in step S51 and the AOT mask generated in step S52 to generate a combined mask of the propagation destination image 81, and supplies it to the zoom unit 144.

[0106] In step S54, the zoom unit 144 extracts the FoV region based on the combined mask generated in step S53. In step S55, the zoom unit 144 extracts a number of points from the combined mask generated in step S53 to be used as prompts in SAM2.

[0107] In step S56, the improvement unit 145 generates an improved mask for the FoV region extracted in step S53 based on the prompt extracted in step S54. The improvement unit 145 supplies the mask region information of the improved mask to the display unit 106.

[0108] In step S57, the improvement unit 145 supplies the mask information of the improved mask generated in step S56 to the SAM2 memory bank 104a and the AOT memory bank 104b, and stores it as the mask information of the destination image 81.

[0109] In step S58, the time-direction propagation unit 103 determines whether or not it has determined all images 81 other than the frame of the image to be operated, which were captured by the reference camera, to be the destination image 81.

[0110] If it is determined in step S58 that the images 81 of all frames other than the frame of the image to be manipulated have not yet been determined as the destination image 81, the process proceeds to step S59.

[0111] In step S59, the time-direction propagation unit 103 determines the image 81 of the next frame in the propagation direction from the current destination image 81, which is captured by the reference camera, as the destination image 81. If there is no image 81 in the propagation direction from the next frame from the current destination image 81, that is, if the current destination image 81 is the last frame in the propagation direction, the time-direction propagation unit 103 changes the propagation direction to the opposite direction. Then, the time-direction propagation unit 103 determines the image 81 of the next frame from the target image in that propagation direction as the destination image 81. After the processing in step S59, the process returns to step S51, and the subsequent processing is repeated.

[0112] If it is determined in step S58 that all frames of the image other than the frame of the target image have been selected as the destination image 81, the time-direction propagation process ends, and the process returns to the process in step S12 in Figure 7, and proceeds to the process in step S13.

[0113] <Example of the result of time propagation processing> Figure 19 shows an example of a segmentation mask obtained as a result of time propagation processing, as displayed in step S13 of Figure 7.

[0114] In the example shown in Figure 19, the time propagation process yields segmentation masks 321 to 324 for each of the four images 81 other than the target image, which were captured by the reference camera. In this case, the display unit 106 highlights the segmentation masks 321 to 324 of each image 81, as shown in Figure 19. Note that in Figure 19, the highlighting is represented by diagonal lines.

[0115] In the example shown in Figure 19, the user checks the segmentation masks 321 to 324 displayed on the display unit 106 and determines that the segmentation masks 321 to 323, which include only the sword, are correct, while the segmentation mask 324, which includes the upper body of a person in addition to the sword, is incorrect. The user then performs an operation to input that the segmentation mask 324 is incorrect. The operation unit 101 receives this operation and adds an error flag to the mask information of the segmentation mask 324 stored in the SAM2 memory bank 104a and the AOT memory bank 104b.

[0116] <Example of the configuration of the spatial directional propagation unit> Figure 20 is a block diagram showing an example of the configuration of the spatial directional propagation unit 105.

[0117] The spatial direction propagation unit 105 in Figure 20 includes a camera classification unit 301, a selection unit 302, a spatial direction prediction unit 303, a selection unit 304, a visual hull generation unit 305, a silhouette generation unit 306, a zoom unit 307, and a spatial direction prediction unit 308.

[0118] The camera classification unit 301 groups the 31 cameras 71 together based on their external parameters, i.e., cameras 71 with similar camera angles. Specifically, the camera classification unit 301 determines the group of each camera 71 based on its external parameters.

[0119] The external parameters of camera 71 are parameters that represent the position and orientation of camera 71. Since camera 71 is placed in the volumetric capture studio 70, the external parameters of camera 71 can be easily obtained. The camera classification unit 301 supplies group information to the selection unit 302, which indicates one or more groups of cameras 71 obtained as a result of grouping, and the cameras 71 belonging to that group arranged according to predetermined rules.

[0120] The selection unit 302 selects group information for the reference group, which includes the reference camera, from the group information supplied by the camera classification unit 301, and supplies it to the spatial direction prediction unit 303. The selection unit 302 supplies group information for groups other than the reference group to the spatial direction prediction unit 308.

[0121] The spatial direction prediction unit 303 (same group prediction unit) determines, based on group information, for each frame, the same group images, which are images 81 taken by the same group cameras (same group imaging devices) other than the reference camera belonging to the reference group, in order to be the target image 81 for propagation. For each frame, the spatial direction prediction unit 303 performs a same group spatial direction propagation process in which the segmentation mask of the image 81 taken by the reference camera is propagated in the spatial direction using a dual-model self-reinforced memory algorithm.

[0122] The co-spatial spatial propagation process is the same as the temporal propagation process, except that the propagation direction is spatial rather than temporal, so its explanation will be omitted. In the co-spatial spatial propagation process, the reference image for each frame is basically the image 81 of that frame, captured by the reference camera. However, if an error flag is attached to the mask information of the segmentation mask of that image 81, then the images 81 of the frames before and after that image 81 will be used as reference images.

[0123] The spatial direction prediction unit 303 obtains the SAM2 mask, AOT mask, combined mask, and improved mask for each associated image by performing a co-sequence spatial direction propagation process for each frame. The spatial direction prediction unit 303 calculates the similarity between the SAM2 mask and AOT mask of each associated image for each frame, i.e., the degree of model independence. Here, since the overlapping region of the SAM2 mask and AOT mask is emphasized, the Dice coefficient is adopted as this similarity. Specifically, the spatial direction prediction unit 303 calculates the Dice coefficient D using the following equation (1).

[0124]

[0125] In equation (1), A is the SAM2 mask and B is the AOT mask. According to equation (1), the Dice coefficient D is the ratio of the overlapping region, which is the region included in both the SAM2 mask and the AOT mask, to half the combined region, which is the region included in at least one of the SAM2 mask and the AOT mask.

[0126] The spatial direction prediction unit 303 calculates a camera score by normalizing and adding the dice coefficient D and the improvement mask for each frame and for each related image. A higher camera score corresponds to a larger dice coefficient D and improvement mask. In other words, a higher camera score indicates higher prediction reliability and clearer depiction of object features and context included in the improvement mask. The spatial direction prediction unit 303 supplies the camera score and improvement mask for each related image to the selection unit 304 for each frame.

[0127] The spatial direction prediction unit 303 supplies the mask region information of the improvement mask for each related image to the display unit 106 as the mask region information of the segmentation mask for that related image, for each frame. The spatial direction prediction unit 303 supplies the mask information of the improvement mask for each related image to the SAM2 memory bank 104a and the AOT memory bank 104b as the mask information of the segmentation mask for that related image, for each frame, and stores them.

[0128] Furthermore, if the dice coefficient D is greater than or equal to a predetermined threshold (for example, 0.7), the spatial direction prediction unit 303 may determine that the SAM2 mask and the AOT mask are similar and add a confidence flag to the corresponding segmentation mask to indicate confidence. Specifically, the spatial direction prediction unit 303 may add a confidence flag to the segmentation masks of similar images stored in the SAM2 memory bank 104a and the AOT memory bank 104b, where the dice coefficient D is greater than or equal to the threshold.

[0129] The selection unit 304 (same-group prediction result selection unit) selects three improvement masks from among the improvement masks of the same-group images in each frame, based on the camera score supplied from the spatial direction prediction unit 303, starting with the ones with the highest camera scores. Note that the number of improvement masks selected is not limited to three. The selection unit 304 supplies the three improvement masks selected for each frame to the visual hull generation unit 305.

[0130] The visual hull generation unit 305 generates a 3D visual hull for each frame from the three improved masks supplied from the selection unit 304, based on information about the viewpoints of three related cameras corresponding to those masks, using the viewing volume cross-eyed method. The information about the viewpoints of the related cameras is generated based on the external and internal parameters of those cameras. The internal parameters of the related cameras can be easily obtained, similar to the external parameters. The visual hull generation unit 305 supplies the generated 3D visual hull to the silhouette generation unit 306.

[0131] The silhouette generation unit 306 generates a 2D silhouette from the viewpoint of each camera 71 by projecting the 3D visual hull supplied from the visual hull generation unit 305 in reverse onto the viewpoint of each camera 71 for each frame.

[0132] The visual hull used to generate the silhouette is a low-precision visual hull generated from improved masks of three identical cameras. Therefore, the accuracy of the silhouette does not meet the accuracy required for segmentation, but it is accurate enough to show the approximate position and shape of the segmentation mask. The silhouette generation unit 306 supplies the silhouettes of each frame of each camera 71 to the zoom unit 307.

[0133] The zoom unit 307 zooms in on the image 81 by extracting the FoV region based on the silhouette supplied from the silhouette generation unit 306 for each camera 71 and for each frame, similar to the zoom unit 144, and generates a zoomed image. The zoom unit 307 generates a zoom mask for the segmentation mask in the zoomed image of each camera 71 belonging to the reference group. The zoom unit 307 supplies the mask information of the generated zoomed image and zoom mask to the SAM2 memory bank 104a and the AOT memory bank 104b for storage.

[0134] The spatial direction prediction unit 308 (different group prediction unit) determines, based on group information, for each frame, the zoomed image of the different group image 81, which is an image captured by a different group camera (different group imaging device) that is a camera 71 belonging to a different group from the reference camera, as the target image 81 for propagation. For each frame, the spatial direction prediction unit 308 performs different group spatial direction propagation processing to propagate the zoom mask in the spatial direction using a dual-model self-reinforced memory algorithm.

[0135] The spatial propagation process for different spatial regions is the same as the spatial propagation process for the same spatial region, except that the reference image is a zoomed-in image of image 81 taken by the reference camera, and the propagation target is a zoom mask. Therefore, no explanation is provided.

[0136] The spatial direction prediction unit 308 zooms out the improved mask of the zoomed-in image of the alien image obtained as a result of the alien spatial direction propagation processing, and uses it as the final segmentation mask of the alien image. The spatial direction prediction unit 308 supplies the mask region information of the segmentation mask of the alien image to the display unit 106. The spatial direction prediction unit 308 supplies the mask information of the segmentation mask of the alien image to the SAM2 memory bank 104a and the AOT memory bank 104b for storage.

[0137] As described above, in the inter-spatial directional propagation process, a zoomed image is used as the reference image, which prevents the mask size ratio from becoming too small. As a result, the segmentation mask can be predicted without being affected by the overall differences in image 81 due to differences in camera angles, and only the image 81 near the segmentation mask is affected. Consequently, prediction performance can be improved.

[0138] Furthermore, the co-space directional propagation process and the dis-space directional propagation process do not necessarily require the use of a dual-model self-reinforcing memory algorithm.

[0139] <Example of Reference Image Selection in Co-spatial Directional Propagation Processing> Figure 21 shows an example of reference image selection in co-spatial directional propagation processing.

[0140] In the example shown in Figure 21, the reference camera captures four frames of images 81, sequentially from left to right in Figure 21, and the mask information of the segmentation masks 341 to 344 for each image 81 is stored in the SAM2 memory bank 104a and the AOT memory bank 104b.

[0141] In this case, the spatial direction prediction unit 303 determines whether or not an error flag has been added to the mask information of the segmentation masks 341 to 344 stored in the SAM2 memory bank 104a and the AOT memory bank 104b. In the example in Figure 21, no error flags have been added to any of the segmentation masks 341 to 344.

[0142] The spatial direction prediction unit 303 also determines, based on the segmentation masks 341 to 344 and the camera angle of the reference camera, whether or not occlusion of a predetermined size or larger occurs in each of the segmentation masks 341 to 344. In the example in Figure 21, segmentation masks 342 and 343 are obscured by a person, resulting in occlusion.

[0143] The spatial direction prediction unit 303 further determines whether the size of the segmentation masks 341 to 344 is smaller than a predetermined threshold. In the example in Figure 21, the size of the segmentation mask 344 is smaller than the threshold.

[0144] Based on the above, the spatial direction prediction unit 303 selects the image 81 of the first frame as the reference image for the second to fourth frames from the beginning. Specifically, the spatial direction prediction unit 303 selects the image 81 of the nearest first frame that includes a segmentation mask 341 in which the corresponding mask information does not have an error flag attached, no occlusion occurs, and has a size greater than or equal to a predetermined threshold.

[0145] As described above, the spatial direction prediction unit 303 selects a reference image for each frame based on the presence or absence of an error flag corresponding to the image 81 of each frame captured by the reference camera, the presence or absence of occlusion, and the size of the segmentation mask.

[0146] In other words, the spatial direction prediction unit 303 selects an appropriate reference image from the images 81 captured by the reference camera for each frame. Therefore, the prediction performance of the same spatial direction propagation process can be improved. Since this selection of reference images is performed by the spatial direction prediction unit 303, the user does not need to perform annotation by selecting an appropriate reference image for each frame.

[0147] <Examples of images of the same frame taken by different cameras> Figure 22 shows examples of images of the same frame, specifically image 81-0 taken by camera 71-0 and image 81-7 taken by camera 71-7.

[0148] Because the camera angles of camera 71-0 and camera 71-7 are not similar, even though images 81-0 and 81-7 are from the same frame, the size and appearance of the subject differ greatly, as shown in Figure 22, resulting in inconsistency. Therefore, propagation of the segmentation mask in the spatial direction is more difficult than propagation in the temporal direction.

[0149] Therefore, the spatial propagation unit 105 groups the 31 cameras 71 together based on their similar camera angles, and first performs a spatial propagation process for the same group. This enables high-precision propagation of the spatial segmentation mask to the same group images.

[0150] <Examples of camera images from each group> Figure 23 shows examples of images 81 from camera 71 of each group.

[0151] In the example shown in Figure 31, the camera classification unit 301 classifies camera 71-0, which is located on the ceiling of the volumetric capture studio 70, into group #0, which is the 0th group. The camera classification unit 301 classifies cameras 71-1 to 71-6, which are located on the upper level of the volumetric capture studio 70, into group #1, which is the 1st group.

[0152] The camera classification unit 301 classifies cameras 71-7 to 71-18, located in the middle section of the volumetric capture studio 70, into the second group, group #2. The camera classification unit 301 classifies cameras 71-19 to 71-30, located in the lower section of the volumetric capture studio 70, into the third group, group #3.

[0153] In this case, as shown in Figure 23, the images 81-1 to 81-6 taken by cameras 71-1 to 71-6 belonging to group #1 show no significant changes in the size or appearance of the subject and are consistent. The same is true for the images 81-7 to 81-18 taken by cameras 71-7 to 71-18 belonging to group #2. Although not shown in the illustration, the same is true for the images 81-19 to 81-30 taken by cameras 71-19 to 71-30 belonging to group #3.

[0154] <Example of the result of co-spatial spatial direction propagation processing> Figure 24 shows an example of the result of co-spatial spatial direction propagation processing by the spatial direction prediction unit 303.

[0155] In the example in Figure 24, cameras 71-7 to 71-18 are grouped into group #2 and arranged in order, similar to the example in Figure 23. In the example in Figure 24, the reference image is image 81-9.

[0156] In this case, as shown in Figure 24, the spatial direction prediction unit 303 predicts the segmentation mask 362 of image 81-8 of camera 71-8, which is in the order preceding camera 71-9, based on the segmentation mask 361 of image 81-9. Subsequently, the spatial direction prediction unit 303 predicts the segmentation mask 363 of image 81-7 of camera 71-7, which is in the order preceding camera 71-8, based on the segmentation mask 362, etc.

[0157] Furthermore, the spatial direction prediction unit 303 predicts the segmentation mask 364 of image 81-10 of camera 71-10, which is in the order of camera 71-9, based on the segmentation mask 361. Subsequently, the spatial direction prediction unit 303 predicts the segmentation mask 365 of image 81-11 of camera 71-11, which is in the order of camera 71-11, based on the segmentation mask 364, etc. Although not shown in the diagram, the spatial direction prediction unit 303 then sequentially predicts the segmentation masks of images 81-12 to 81-18 of cameras 71-12 to 71-18, which are in the order of camera 71-11.

[0158] <Examples of SAM2 masks and AOT masks when the camera score is high> Figure 25 shows examples of SAM2 masks and AOT masks when the camera score is high.

[0159] When the camera score is high, the SAM2 mask 381 and AOT mask 382 of the image 81 corresponding to that camera score are similar, as shown in Figure 25. Therefore, it is highly likely that the SAM2 mask 381 and AOT mask 382 are correct. Thus, the accuracy of the improved mask generated using the SAM2 mask 381 and AOT mask 382 is high.

[0160] Based on the above, the selection unit 304 selects, for each frame, the three improvement masks with the highest camera scores from among the improvement masks of each corresponding camera in that frame, as improvement masks to be used for generating the visual hull.

[0161] <Example of a silhouette> Figure 26 shows an example of a silhouette generated by the silhouette generation unit 306.

[0162] In the example shown in Figure 26, a silhouette 401 representing the viewpoint of camera 71-0 and a silhouette 402 representing the viewpoint of camera 71-7 are generated from a 3D visual hull of a predetermined frame. In Figure 26, silhouette 401 is displayed superimposed on image 81-0 of that frame, and silhouette 402 is displayed superimposed on image 81-7 of that frame.

[0163] As shown in Figure 26, when silhouette 401 (402) is small relative to the entire image 81-0 (81-7), it is difficult for the user to visually identify silhouette 401 (402).

[0164] <Example of a zoomed image> Figure 27 shows an example of a zoomed image generated by the zoom unit 307.

[0165] As shown in Figure 27A, each of the zoom images 421 to 423 from the three cameras 71 belonging to the reference group is an image of the vicinity of the respective segmentation masks 421a to 423a in the image 81 captured by each camera 71.

[0166] As shown in Figure 27B, the zoom images 441 and 442 from the two different cameras are images of the vicinity of the region in the different image captured by that camera that is presumed to contain the segmentation mask. Specifically, in the example in Figure 27, the object included in the segmentation masks 421a to 423a is a sword, and the zoom images 441 and 442 are images of the vicinity of the region containing the sword in the different image.

[0167] As described above, zoom images 421-423 and 441 and 442 are images of the area around the sword in image 81. Therefore, in the spatial direction prediction unit 308, by using the mask information corresponding to zoom images 421-423 and zoom images 441 and 442 in the dissimilar spatial direction propagation processing, it is possible to prevent the unit from being affected by areas other than the sword that are unnecessary for segmentation. In other words, the spatial direction prediction unit 308 can extract the features of the sword and track them in the spatial direction. Therefore, it is possible to propagate the spatial direction segmentation mask to dissimilar images where the size and appearance of the subject differ due to the dissimilar camera angles with high accuracy.

[0168] <Explanation of spatial directional propagation processing> Figure 28 is a flowchart illustrating the spatial directional propagation processing performed by the spatial directional propagation unit 105 in Figure 20.

[0169] In step S70 of Figure 28, the camera classification unit 301 groups the 31 cameras 71 based on their external parameters. The camera classification unit 301 supplies group information for each group to the selection unit 302.

[0170] In step S71, the selection unit 302 selects the group information of the reference group from the group information supplied by the camera classification unit 301 and supplies it to the spatial direction prediction unit 303. The selection unit 302 also selects the group information of groups other than the reference group and supplies it to the spatial direction prediction unit 308.

[0171] In step S72, the spatial direction prediction unit 303 determines, frame by frame, the corresponding images to be the destination image 81 based on the group information supplied from the selection unit 302. In step S73, the spatial direction prediction unit 303 performs corresponding spatial direction propagation processing for each frame for the destination image 81 determined in step S72.

[0172] In step S74, the spatial direction prediction unit 303 calculates the camera score for each co-image based on the SAM2 mask, AOT mask, and improvement mask of each co-image obtained as a result of the co-image spatial direction propagation processing in step S73, for each frame. The spatial direction prediction unit 303 supplies the camera score and improvement mask of each co-image to the selection unit 304 for each frame. The spatial direction prediction unit 303 supplies the mask region information of the improvement mask of each co-image to the display unit 106 as the mask region information of the segmentation mask of that co-image, for each frame. The spatial direction prediction unit 303 supplies the mask information of the improvement mask of each co-image to the SAM2 memory bank 104a and the AOT memory bank 104b for storage, for each frame, as the mask information of the segmentation mask of that co-image.

[0173] In step S75, the selection unit 304 selects three improvement masks for each frame, based on the camera score calculated in step S74, from among the improvement masks of each related image in that frame, starting with the ones with the highest camera scores. The selection unit 304 supplies the three improvement masks selected for each frame to the visual hull generation unit 305.

[0174] In step S76, the visual hull generation unit 305 generates a 3D visual hull for each frame based on information about the viewpoints of three identical cameras corresponding to the three improved masks selected in step S75. The visual hull generation unit 305 supplies the generated visual hull to the silhouette generation unit 306.

[0175] In step S77, the silhouette generation unit 306 generates a 2D silhouette of the viewpoint of each camera 71 from the 3D visual hull generated in step S76 for each frame. The silhouette generation unit 306 supplies the silhouette of each frame of each camera 71 to the zoom unit 307.

[0176] In step S78, the zoom unit 307 generates a zoom image for each camera 71 and for each frame based on the silhouette generated in step S77, and generates a zoom mask for each frame of each camera 71 belonging to the reference group. The zoom unit 307 supplies the mask information of the generated zoom image and zoom mask to the SAM2 memory bank 104a and the AOT memory bank 104b for storage.

[0177] In step S79, the spatial direction prediction unit 308, based on the group information supplied from the selection unit 302, sequentially determines the zoomed images of different images as the target image 81 for each frame.

[0178] In step S80, the spatial direction prediction unit 308 performs a different spatial direction propagation process for each frame for the propagation destination image 81 determined in step S79. The spatial direction prediction unit 308 zooms out the improved mask of the zoomed-in image of the different image obtained as a result of the different spatial direction propagation process, and uses it as the final segmentation mask of the different image. The spatial direction prediction unit 308 supplies the mask region information of the segmentation mask of the different image to the display unit 106. The spatial direction prediction unit 308 supplies the mask information of the segmentation mask of the different image to the SAM2 memory bank 104a and the AOT memory bank 104b for storage. Then the spatial direction propagation process is completed, and the process returns to step S15 in Figure 7, and the segmentation process is completed.

[0179] Based on the spatial propagation processing shown in Figure 28, the display unit 106 displays the segmentation masks for each related and disrelated image in each frame, based on the mask region information supplied to the display unit 106. The user performs an operation to input whether the segmentation mask is correct or incorrect. Based on this correctness, the operation unit 101 adds an error flag to the segmentation masks for each related and disrelated image in each frame, stored in the SAM2 memory bank 104a and the AOT memory bank 104b.

[0180] As described above, in the information processing system 100, the SAM2 prediction unit 141 predicts the segmentation mask of the destination image 81, which is captured by the reference camera at a different time than the target image, using SAM2, based on the mask information of the target image. The AOT prediction unit 142 predicts the segmentation mask of the destination image 81 using AOT, based on the mask information of the target image. Then, the zoom unit 144 extracts the FoV region of the destination image 81 based on the SAM2 mask obtained as a result of the prediction by the SAM2 prediction unit 141 and the AOT mask obtained as a result of the prediction by the AOT prediction unit 142. The improvement unit 145 predicts the segmentation mask of the destination image 81 using SAM2 based on the FoV region and obtains the final prediction result.

[0181] Therefore, the information processing system 100 can accurately segment the video footage captured by the reference camera while reducing the user and processing load. Specifically, the user can accurately segment the video footage captured by the reference camera simply by inputting prompts for the target image.

[0182] Furthermore, the spatial direction prediction unit 303 predicts the segmentation mask in the same-type images based on the segmentation mask of the image 81 of each frame captured by the reference camera. The zoom unit 307 extracts parts of the same-type and different-type images as zoomed images based on the prediction results by the spatial direction prediction unit 303. The spatial direction prediction unit 308 predicts the segmentation mask in the different-type images based on these zoomed images.

[0183] Therefore, the information processing system 100 can accurately segment video footage captured by cameras 71 other than the reference camera, while reducing the user and processing load. Specifically, the user can accurately segment video footage captured by cameras 71 other than the reference camera without performing additional training, fine tuning, or additional user operations.

[0184] As described above, the user can accurately segment moving images captured by multiple cameras 71 simply by inputting prompts for the target image. In contrast, if the segmentation mask can only be propagated in the time direction, the user and processing load will increase by the number of cameras. For example, in segmentation using SAM2 alone, the user needs to input prompts at least as many times as there are cameras.

[0185] Furthermore, the prediction method in the dual-model self-reinforced memory algorithm is not limited to the method using SAM2 (first method) and the method using AOT (second method), but may also be a method using two different machine learning models other than SAM2 and AOT. The machine learning model used for prediction in the improvement unit 145 is not limited to SAM2. The information processing system 100 and the 31 cameras 81 may be integrated.

[0186] <4. Description of a computer to which this technology is applied> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on the computer. Here, the term "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0187] Figure 29 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0188] In a computer, the processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0189] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0190] The input unit 906 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0191] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907 via the bus 904 and the input / output interface 905 as needed. The processing circuit 901 can also store the processing results in the storage unit 908 or transmit them from the communication unit 909.

[0192] The program executed by the computer (processing circuit 901) can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0193] In a computer, a program can be installed in the storage unit 908 via the input / output interface 905 by inserting a removable media 911 into the drive 910. Alternatively, a program can be received by the communication unit 909 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0194] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0195] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0196] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0197] When the computer executes a program to perform the series of processes described above, the input unit 906 functions as the operation unit 101, and the storage unit 908 functions as the memory bank 104. The processing circuit 901 (its processor) functions as the segmentation unit 102, the time propagation unit 103, and the spatial propagation unit 105 by executing the program, and the output unit 907 functions as the display unit 106.

[0198] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory (for example, only the processing circuit 901, or a combination of the processing circuit 901 to the bus 904, etc.) are systems. In the case of a collection of multiple components, it is not necessary whether all components are in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device in which multiple modules are housed in one enclosure, are all systems. Also, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), are systems.

[0199] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0200] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0201] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0202] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0203] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0204] The effects described herein are merely illustrative and not limited to those described herein; other effects may also occur.

[0205] This technology can take the following configurations: (1) An information processing system comprising: a first prediction unit that predicts a predetermined region in a second image taken at a different time from the first image by a camera that took the first image, based on a predetermined region in a first image, using a first method; a second prediction unit that predicts a predetermined region in a second image using a second method, based on the predetermined region in the first image; an extraction unit that extracts a part of the second image based on the prediction results of the first prediction unit and the prediction results of the second prediction unit; and an improvement unit that predicts the predetermined region in the second image using a third method and obtains a final prediction result based on the part of the second image extracted by the extraction unit. (2) The information processing system according to (1) above, configured such that the machine learning models used in the first method and the second method are different. (3) The information processing system according to (1) or (2) above, configured such that the machine learning models used in the first method and the third method are the same. (4) An information processing system according to any one of (1) to (3), further comprising a region extraction unit that extracts a predetermined region within the first image based on user input to the first image displayed on a display unit, wherein the first prediction unit and the second prediction unit make predictions based on the predetermined region within the first image extracted by the region extraction unit. (5) An information processing system according to any one of (1) to (4), wherein the extraction unit is configured to extract a region obtained by combining the prediction result from the first prediction unit and the prediction result from the second prediction unit as part of the second image. (6) An information processing system according to any one of (1) to (5), further comprising a spatial direction prediction unit that predicts the predetermined region in an image taken by a different imaging device than the reference imaging device which is the imaging device that took the first image and the second image, based on the final prediction result obtained from the improvement unit.(7) The information processing system according to (6), wherein the improvement unit causes the final prediction result to be displayed on the display unit, and the spatial direction prediction unit selects a predetermined final prediction result from the final prediction result obtained from the improvement unit based on the user's input to the final prediction result displayed on the display unit, and makes a prediction based on the predetermined final prediction result. (8) The information processing system according to (6), wherein the spatial direction prediction unit selects a predetermined final prediction result from the final prediction result obtained from the improvement unit based on the final prediction result obtained from the improvement unit, and makes a prediction based on the predetermined final prediction result. (9) The information processing system according to any one of (6) to (8), wherein the spatial direction prediction unit comprises: a related prediction unit that predicts the predetermined region in a related image, which is an image taken by another related imaging device that is another imaging device belonging to the same group as the reference imaging device, based on the final prediction result; a zoom unit that extracts a part of a different image, which is an image taken by a different imaging device that is an imaging device belonging to a different group from the reference imaging device, and a part of the related image, as zoom images, based on the prediction result by the related prediction unit; and a different prediction unit that predicts the predetermined region in the different image based on the zoom image extracted by the zoom unit. (10) The information processing system according to (9), wherein the group of imaging devices is configured to be determined based on external parameters of the imaging devices. (11) The information processing system according to (9) or (10), wherein the spatial direction prediction unit further comprises a visual hull generation unit that generates a 3D visual hull based on the prediction result by the related prediction unit, and the zoom unit extracts the zoom image based on the visual hull generated by the visual hull generation unit.(12) The spatial direction prediction unit further comprises a silhouette generation unit that generates 2D silhouettes of the viewpoints of the same-type imaging device and the different-type imaging device based on the visual hull generated by the visual hull generation unit, and the zoom unit extracts the zoom image based on the silhouette generated by the silhouette generation unit, as described in (11). (13) The spatial direction prediction unit further comprises a same-type prediction result selection unit that selects the prediction result from the same-type prediction unit, the number of the same-type imaging devices is multiple, the same-type prediction result selection unit selects the prediction result in the same-type image captured by a predetermined same-type imaging device among the multiple same-type imaging devices based on the prediction result from the same-type prediction unit, and the zoom unit extracts the zoom image based on the prediction result selected by the same-type prediction result selection unit, as described in any of (9) to (12). (14) The information processing system according to any of (1) to (13) configured so that the first image and the second image are acquired from the imaging device. (15) The information processing system according to any one of (1) to (13) above, further comprising the imaging device. (16) An information processing method comprising: predicting a predetermined region in a first image using a first method and a second method, respectively, in a second image taken at a different time from the first image by the imaging device that took the first image, based on a predetermined region in the first image; extracting a part of the second image based on the prediction result by the first method and the prediction result by the second method; and predicting the predetermined region in the second image using a third method based on a part of the second image to obtain a final prediction result.(17) A program for causing a computer to perform a process that includes predicting a predetermined region in a second image taken at a different time from the first image by a camera that took the first image, based on a predetermined region in the first image, using a first method and a second method, respectively; extracting a part of the second image based on the prediction result by the first method and the prediction result by the second method; and predicting the predetermined region in the second image using a third method based on the part of the second image, and obtaining a final prediction result.

[0206] 71-1 to 71-30 Camera, 81-1 to 81-30 Image, 100 Information processing system, 102 Segmentation unit, 105 Spatial direction propagation unit, 141 SAM2 prediction unit, 142 AOT prediction unit, 144 Zoom unit, 145 Improvement unit, 303 Spatial direction prediction unit, 304 Selection unit, 305 Visual hull generation unit, 306 Silhouette generation unit, 307 Zoom unit, 308 Spatial direction prediction unit

Claims

1. An information processing system comprising: a first prediction unit that predicts a predetermined region in a second image taken at a different time from the first image by the same imaging device that took the first image, based on a predetermined region in a first image, using a first method; a second prediction unit that predicts a predetermined region in a second image using a second method, based on the predetermined region in the first image; an extraction unit that extracts a portion of the second image based on the prediction results from the first and second prediction units; and an improvement unit that predicts the predetermined region in the second image using a third method and obtains a final prediction result based on the portion of the second image extracted by the extraction unit.

2. The information processing system according to claim 1, wherein the machine learning models used in the first method and the second method are configured to be different.

3. The information processing system according to claim 1, configured such that the machine learning models used in the first method and the third method are identical.

4. The information processing system according to claim 1, further comprising: a region extraction unit that extracts a predetermined region within the first image based on user input to the first image displayed on a display unit, wherein the first prediction unit and the second prediction unit perform predictions based on the predetermined region within the first image extracted by the region extraction unit.

5. The information processing system according to claim 1, wherein the extraction unit is configured to extract a region obtained by combining the prediction result from the first prediction unit and the prediction result from the second prediction unit as part of the second image.

6. The information processing system according to claim 1, further comprising a spatial direction prediction unit that predicts the predetermined region in an image captured by a different imaging device than the reference imaging device which is the imaging device that captured the first image and the second image, based on the final prediction result obtained from the improvement unit.

7. The information processing system according to claim 6, wherein the improvement unit displays the final prediction result on the display unit, and the spatial direction prediction unit selects a predetermined final prediction result from the final prediction result obtained from the improvement unit based on the user's input to the final prediction result displayed on the display unit, and performs a prediction based on the predetermined final prediction result.

8. The information processing system according to claim 6, wherein the spatial direction prediction unit selects a predetermined final prediction result from the final prediction results obtained from the improvement unit based on the final prediction results obtained from the improvement unit, and performs a prediction based on the predetermined final prediction result.

9. The information processing system according to claim 6, comprising: a spatial direction prediction unit, a related prediction unit that predicts a predetermined region in a related image, which is an image taken by another related imaging device that belongs to the same group as the reference imaging device, based on the final prediction result; a zoom unit that extracts a part of a different image, which is an image taken by a different imaging device that belongs to a different group from the reference imaging device, and a part of the related image, as zoom images, based on the prediction result by the related prediction unit; and a different prediction unit that predicts a predetermined region in the different image, based on the zoom images extracted by the zoom unit.

10. The information processing system according to claim 9, wherein the group of imaging devices is configured to be determined based on external parameters of the imaging devices.

11. The information processing system according to claim 9, wherein the spatial direction prediction unit further comprises a visual hull generation unit that generates a 3D visual hull based on the prediction results of the spatial direction prediction unit, and the zoom unit extracts the zoom image based on the visual hull generated by the visual hull generation unit.

12. The information processing system according to claim 11, wherein the spatial direction prediction unit further comprises a silhouette generation unit that generates 2D silhouettes of the viewpoints of the same-type imaging device and the different-type imaging device based on the visual hull generated by the visual hull generation unit, and the zoom unit extracts the zoom image based on the silhouette generated by the silhouette generation unit.

13. The information processing system according to claim 9, wherein the spatial direction prediction unit further comprises a same-type prediction result selection unit that selects the prediction result from the same-type prediction unit, the number of the same-type imaging devices is multiple, the same-type prediction result selection unit selects the prediction result from the same-type image captured by a predetermined same-type imaging device among the multiple same-type imaging devices based on the prediction result from the same-type prediction unit, and the zoom unit extracts the zoom image based on the prediction result selected by the same-type prediction result selection unit.

14. The information processing system according to claim 1, configured such that the first image and the second image are acquired from the imaging device.

15. The information processing system according to claim 1, further comprising the aforementioned imaging device.

16. An information processing method comprising: an information processing system predicting a predetermined region in a second image taken at a different time from the first image by a camera that took the first image, based on a predetermined region in a first image, using a first method and a second method, respectively; extracting a part of the second image based on the prediction results of the first method and the second method; and predicting the predetermined region in the second image using a third method based on the part of the second image, and obtaining a final prediction result.

17. A program for causing a computer to perform a process that includes predicting a predetermined region in a second image, taken at a different time from the first image by the same camera that took the first image, based on a predetermined region in the first image, using a first method and a second method, respectively; extracting a portion of the second image based on the prediction results of the first method and the second method; and predicting the predetermined region in the second image using a third method based on the portion of the second image, thereby obtaining a final prediction result.

Images