Image processing device and display control method

Abstract

In the present invention, an acquisition unit acquires a plurality of frame images including a target frame image. A determination unit determines a region of a first object in the target frame image on the basis of the degree of similarity between the target frame image and at least one frame image included in the plurality of frame images. A display control unit controls a display device to display an image indicating the region of the first object in the target frame image.

Description

Image Processing Apparatus and Display Control Method

[0001] The present disclosure relates to an image processing apparatus and a display control method.

[0002] Conventionally, semantic segmentation technology for estimating the area occupied by an object (such as an object) in an image is known. For example, Non-Patent Document 1 below discloses a technique for realizing semantic segmentation in an entire video by specifying one prompt (information specifying an object) for the video. With the technique of Non-Patent Document 1, it is possible to estimate the area occupied by an object at the pixel level without requiring a large amount of learning data.

[0003] Nikhila Ravi et al., "SAM 2: Segment Anything in Images and Videos", [online], July 2024, meta, [searched on October 17, 2024], Internet, <URL: https: / / ai.meta.com / research / publications / sam-2-segment-anything-in-images-and-videos / >

[0004] In the above-described conventional technology, among the plurality of frame images constituting the video, the analysis result (the estimation result of the area occupied by the object) in a predetermined number of frame images immediately before the frame image to be processed is used for the analysis of the frame image to be processed. Therefore, when there is a frame image in which no object appears in the video, there is a problem that the accuracy of the estimation result decreases. Examples of the case where there is a frame image in which no object appears include, for example, when a video (or still image) different from the main video, such as a commercial film, is inserted into the main video, or when the object temporarily disappears from the screen and then reappears in the screen again.

[0005] An object of the present disclosure is to improve the accuracy of semantic segmentation for an image.

[0006] The image processing apparatus according to this disclosure includes: an acquisition unit that acquires a plurality of frame images including a target frame image; a determination unit that determines the region of a first object in the target frame image based on the similarity between the target frame image and at least one frame image included in the plurality of frame images; and a display control unit that controls a display device to display an image showing the region of the first object in the target frame image.

[0007] The display control method according to this disclosure acquires a plurality of frame images including a target frame image, determines the region of a first object in the target frame image based on the similarity between the target frame image and at least one frame image included in the plurality of frame images, and controls the display device to display an image showing the region of the first object in the target frame image.

[0008] According to this disclosure, the accuracy of semantic segmentation of images can be improved.

[0009] This is a block diagram showing the configuration of the image processing device 10 according to the embodiment. This is a block diagram showing the functional configuration of the segmentation model SM. This is a schematic diagram showing the video DG to be analyzed. This is a diagram showing an example of a frame image FM. This is a schematic diagram showing how to determine the mask data MD-1 for the prompt-specified frame image FM-P. This is a schematic diagram showing how to generate memory ME. This is a schematic diagram showing how to determine the mask data MD-T for the target frame image FM-T (the second and subsequent frame images FM). This is a schematic diagram showing the similarity of frame images FM. This is a schematic diagram showing an example of a region display image IR. This is a schematic diagram showing the difference in the method of selecting memory ME between the embodiment and the comparative example. This is a flowchart showing the operation of the processing device 106.

[0010] [Embodiment] [Configuration of Image Processing Device 10] Figure 1 is a block diagram showing the configuration of an image processing device 10 according to an embodiment. The image processing device 10 is an information processing device such as a personal computer, server, smartphone, or tablet terminal. In this embodiment, the image processing device 10 is assumed to be a personal computer.

[0011] In this embodiment, the image processing device 10 performs semantic segmentation on a video and outputs the result. Specifically, semantic segmentation involves segmenting the area occupied by a specified object on the image (hereinafter referred to as "object area RO") and displaying the object area RO in a way that makes it distinguishable from other areas (or outputting information that makes it possible to distinguish the object area RO from other areas). Hereinafter, the video subject to semantic segmentation will be referred to as the "analyzed video DG".

[0012] The image processing device 10 includes a display device 101, an input device 102, a communication device 103, an interface 104, a storage device 105, a processing device 106, and a bus 120 that connects these devices to each other.

[0013] The display device 101 is a display device that displays information to the outside (for example, various display panels such as a liquid crystal display panel or an organic EL display panel). The input device 102 is an input device that accepts input from the outside (for example, a keyboard, mouse, microphone, switch, button, or sensor). The display device 101 and the input device 102 may be configured as an integrated unit (for example, a touch panel).

[0014] The communication device 103 is equipped with a communication interface that can connect to a network (not shown) and communicates with other devices connected to the network using wireless or wired communication. The communication device 103 may perform short-range wireless communication such as Bluetooth® or Wi-Fi®.

[0015] Interface 104 connects the image processing device 10 to an external device. Interface 104 may be, for example, a USB (Universal Serial Bus) port or a memory card slot. In this embodiment, interface 104 and communication device 103 are mainly used to acquire the video DG to be analyzed.

[0016] The storage device 105 is a recording medium that can be read by the processing device 106. The storage device 105 includes, for example, non-volatile memory and volatile memory. Non-volatile memory includes, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory includes, for example, RAM (Random Access Memory).

[0017] The storage device 105 stores the program PG1 and the segmentation model SM. Program PG1 is a program for operating the image processing device 10. The segmentation model SM is a machine learning model that performs segmentation processing based on, for example, SAM (Segment Anything Model) 2, and continuously identifies the region of the target object OT in the video DG to be analyzed, which will be described later.

[0018] Figure 2 is a block diagram showing the functional configuration of the segmentation model SM. The segmentation model SM includes a prompt encoder 21, an image encoder 22, a memory attention 23, a mask decoder 24, a memory decoder 25, and a memory bank 26.

[0019] The prompt encoder 21 converts the prompt PT into a format that can be processed within the model. The image encoder 22 converts the input frame image FM into a format that can be processed within the model. The memory attention 23 reads a predetermined number of memory ME from the memory bank 26. The mask decoder 24 generates mask data MD, which will be described later. The memory decoder 25 generates memory ME, which will be described later. The memory bank 26 stores the memory ME, which will be described later.

[0020] The processing unit 106 includes one or more CPUs (Central Processing Units). One or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0021] The processing unit 106 reads program PG1 from the storage device 105. By executing program PG1, the processing unit 106 functions as an acquisition unit 111, a reception unit 112, a determination unit 113, and a display control unit 114. At least one of these functional units may be composed of circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0022] [Functions of the Processing Unit 106] The acquisition unit 111 acquires the video to be analyzed (hereinafter referred to as "video to be analyzed DG"). If the frame image FM to be processed in the determination unit 113, which will be described later, is the target frame image FM-T, then it can be said that the acquisition unit 111 acquires multiple frame images FM, including the target frame image FM-T.

[0023] The acquisition unit 111 acquires the video to be analyzed DG by receiving the video to be analyzed DG data from another information processing device connected to the network, for example, using a communication device 103. Alternatively, the acquisition unit 111 acquires the video to be analyzed DG by reading the video to be analyzed DG data from a portable memory connected to an interface 104, for example.

[0024] Figure 3 is a schematic diagram of the video DG to be analyzed. The video DG to be analyzed contains multiple frame images FM (FM-1 to FM-N). In the example in Figure 3, the multiple frame images FM are N (N is an integer greater than or equal to 2) frame images FM that make up the video. The N frame images FM are numbered from 1 to N according to the order in which the N frame images FM are arranged in the video. Specifically, the first frame image FM is numbered as frame image FM-1, the Nth frame image FM is numbered as frame image FM-N, and so on. The number assigned to the frame image FM can also be called the frame image FM ID.

[0025] Figure 4 shows an example of frame image FM. Figure 4 shows four frame images FM-1 to FM-4. ​​Frame images FM-1 to FM-4 show a toy train RS, the tracks on which the train RS runs, and a model station building. Frame images FM-1 to FM-4 are arranged chronologically in numerical order, and it can be seen that the train RS moves as time progresses.

[0026] The source for frame images FM-1 to FM-4 is shown below: Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A. and Van Gool, L.: The 2017 Davis Challenge on Video Object Segmentation, arXiv preprint arXiv:1704.00675 (2017).

[0027] The reception unit 112 accepts the designation of a target object OT for the first frame image FM-1 out of N frame images FM. The target object OT is an example of a first object. The target object OT is an object whose region is identified in the video DG to be analyzed.

[0028] The reception unit 112 accepts the designation of a target object OT by receiving a prompt PT input from the user for the frame image FM-1. The prompt PT is, for example, one or more coordinate pieces indicating one or more points on the frame image FM-1. Alternatively, the prompt PT may be four coordinate pieces indicating a rectangular area on the frame image FM-1. Alternatively, the prompt PT may be multiple coordinate pieces indicating an arbitrary area (at least a part of the area occupied by the target object OT).

[0029] Specifically, the reception unit 112 displays the frame image FM-1 on the display device 101, for example. The reception unit 112 instructs the user to select (draw) one or more points within the area where the target object OT is depicted, or to draw a rectangle surrounding the area where the target object OT is depicted (specifically, for example, by displaying instruction text). The user uses the input device 102 to draw the points or rectangle on the frame image FM-1 displayed on the display device 101. The coordinate information of the drawn points or rectangle is acquired as a prompt PT.

[0030] In this embodiment, the input of a prompt PT (specification of the target object OT) is accepted for the first frame image FM-1 of the video DG to be analyzed. However, the invention is not limited to this, and a prompt PT may be input for any frame image FM with an arbitrary ID. In this case, "the first frame image FM-1" in the following description should be read as "the frame image FM for which a prompt PT has been specified".

[0031] The determination unit 113 determines the area of ​​the target object OT (hereinafter referred to as "object area RO") in each of the multiple frame images FM. The determination unit 113 determines the object area RO in each of the N frame images FM in the order of the numbered N frame images FM. That is, for the N frame images FM, the determination unit 113 determines the object area RO in the order of the first frame image FM-1, the second frame image FM-2, ... and finally determines the object area RO in the Nth frame image FM-N.

[0032] The processing of the determination unit 113 will be explained in more detail. The determination unit 113 determines mask data MD that indicates the object region RO in the frame image FM. Mask data MD is binary data (0 or 1) of the same size as the frame image FM. More specifically, for each pixel of the frame image FM, mask data MD takes the value "1" if it is the object region RO, and "0" if it is not the object region RO. The fact that mask data MD and the frame image FM are of the same size means, for example, that if the frame image FM is 256 pixels × 256 pixels, then mask data MD is 256 × 256 binary data.

[0033] Here, the method for determining the mask data MD in the determination unit 113 differs depending on whether the frame image FM is the first frame image FM-1 or the second or later frame image FM. This is because the first frame image FM-1 has received the prompt PT input, while the second and later frame images FM have not. Hereafter, the first frame image FM-1 will be referred to as the prompt-specified frame image FM-P, and the second and later frame images FM will be referred to as the target frame image FM-T. In other words, the target frame image FM-T is the M (where M is an integer between 2 and N)th frame image FM-M.

[0034] Figure 5 is a schematic diagram illustrating the method for determining the mask data MD-1 for the prompt-specified frame image FM-P. The determination unit 113 inputs the prompt-specified frame image FM-P (frame image FM-1) and the prompt PT to the mask decoder 24 of the segmentation model SM. In the following example, all frame images FM, including the prompt-specified frame image FM-P, are assumed to be 256 pixels × 256 pixels, and the prompt PT is the coordinate information (128, 128) of 1 on the prompt-specified frame image FM-P. The coordinate (128, 128) on the prompt-specified frame image FM-P is the pixel in the area where the train RS is captured.

[0035] The segmentation model SM (mask decoder 24) identifies the area of ​​train RS on the prompt-specified frame image FM-P and outputs mask data MD-P in which pixels included in the area of ​​train RS are set to "1" and pixels not included in the area of ​​train RS are set to "0". The mask image IM-P in Figure 5 is an image based on the mask data MD-P, and is an image in which pixels with "1" are drawn in white and pixels with "0" are drawn in black. In other words, the white areas in the mask image IM-P are the object area RO.

[0036] When mask data MD-P is generated, the memory decoder 25 of the segmentation model SM generates memory ME (ME-P). Memory ME is numerical data generated based on the frame image FM and the mask data MD corresponding to the frame image FM. Memory ME is data that represents the feature quantities of the frame image FM and the mask data MD, and is multidimensional (64 × 64 × 64 dimensions in this embodiment) numerical data that is difficult to interpret. In other words, memory ME is an example of feature quantity data.

[0037] Figure 6 schematically illustrates the method for generating memory ME. The frame image FM and mask data MD are input to the memory decoder 25 of the segmentation model SM. At this time, the frame image FM is input to the memory decoder 25 after image features have been extracted by the image encoder 22. The memory decoder 25 outputs memory ME by performing multiple convolution operations on the frame image FM and mask data MD. The memory ME generated by the memory decoder 25 is stored in the memory bank 26. Note that the method for generating memory ME is the same regardless of whether the frame image FM is the prompt-specified frame image FM-P or not.

[0038] Here, the memory ME generated from the prompt-specified frame image FM-P (FM-1) and the mask data MD-P (MD-1) is referred to as the prompt-specified memory ME-P (ME-1).

[0039] Next, the method for generating the mask data MD-T of the second and subsequent frame images FM, i.e., the target frame image FM-T, will be described. The determination unit 113 determines the object region RO in the target frame image FM-T based on the similarity between the target frame image FM-T and at least one frame image FM included in the multiple frame images FM.

[0040] Figure 7 is a schematic diagram illustrating the method for determining the mask data MD-T for the target frame image FM-T (the second and subsequent frame images FM). The determination unit 113 inputs the target frame image FM-T, the prompt specification memory ME-P, and a plurality of similar memory ME-R to the mask decoder 24. The mask decoder 24 determines the mask data MD-T for the target frame image FM-T based on the target frame image FM-T, the prompt specification memory ME-P, and the plurality of similar memory ME-R. The number of similar memory ME-R is arbitrary, but in this embodiment, six similar memory ME-R are used. The number of similar memory ME-R can be, for example, 0 or more and 6 or less.

[0041] Here, the similar memory ME-R is a memory ME corresponding to a similar frame image FM-R having a high similarity to the target frame image FM-T. The determination unit 113 selects six similar frame images FM-R in descending order of similarity to the target frame image FM-T from the frame images FM having an ID smaller than that of the target frame image FM-T. Then, the determination unit 113 reads six memories ME corresponding to the six similar frame images FM-R from the memory bank 26 and inputs them to the mask decoder 24 as the similar memory ME-R.

[0042] For example, when the target frame image FM-T is the M-th frame image FM-M, the determination unit 113 calculates the similarity between the M-th frame image FM-M and the frame images FM from the 1st to the (M - 1)-th. If the 1st frame image FM-1 is the prompt-specified frame image FM-P, the similarity between the 1st frame image FM-1 and the M-th frame image FM-M does not necessarily need to be calculated. This is because the memory ME of the prompt-specified frame image FM-P is used as the prompt-specified memory ME-P for all frame images FM. The similarity between the frame images FM may be calculated using, for example, an AI that calculates image similarity such as AugNet.

[0043] Then, the determination unit 113 selects six frame images FM in descending order of similarity from the frame images FM from the 2nd to the (M - 1)-th as the similar frame images FM-R (not shown), and reads the similar memory ME-R, which is the memory ME of the similar frame image FM-R, from the memory bank 26. The function of this determination unit 113 corresponds to the function of the memory attention 23.

[0044] FIG. 8 is a schematic diagram showing the similarity of the frame image FM. In FIG. 8, images with IDs (denoted as "id" in the figure) from 0 to 4 are shown. The image with ID = 0 corresponds to the prompt-specified frame image FM-P. In the image with ID = 0, the location specified by the prompt PT is indicated by a star mark. Also, the image with ID = 4 corresponds to the target frame image FM-T.

[0045] In the example of FIG. 8, the smaller the similarity S value, the more similar the images are to each other. The similarity S between the image with ID = 4 and the image with ID = 1 is 50. The similarity S between the image with ID = 4 and the image with ID = 2 is 100. The similarity S between the image with ID = 4 and the image with ID = 3 is 5. Therefore, the similarity with the image with ID = 4 is highest for the image with ID = 3, followed by the image with ID = 1, and lowest for the image with ID = 2.

[0046] Note that the calculation of the similarity between the frame images FM may be performed, for example, immediately after the analysis target video DG is acquired. In this case, when the acquisition unit 111 acquires the analysis target video DG, it divides the analysis target video DG into frames and stores them in the image directory (storage device 105) with the name corresponding to the ID of each frame image FM.

[0047] The determination unit 113 uses an AI that calculates the image similarity to calculate the similarity between the calculation target frame image FM-K (for example, frame image FM-6) and other frame images FM (for example, frame images FM-2 to FM-5) having an ID smaller than the ID of the calculation target frame image FM-K. The determination unit 113 stores the calculation result of the similarity in a dictionary format with a list of frame IDs sorted in the order of similarity as the Value, with the ID of each frame image FM as the key. In the example of FIG. 8, {"4": [3, 1, 2, 0, 0, 0]} is stored. Note that when the number of frame images FM to be calculated for similarity is less than 6, the ID of the prompt specified frame image FM-P (ID = 0 in the example of FIG. 8) is inserted into the missing part.

[0048] The determination unit 113 outputs a similarity list of frame IDs with the ID of the frame image FM input as the target frame image FM-T as the key. For example, in the example of FIG. 8, when ID = 4 is input, the similarity list [3, 1, 2, 0, 0, 0] is output. In this case, when determining the mask data MD of the image with ID = 4, the memory ME of the image with ID = 3, the memory ME of the image with ID = 1, the memory ME of the image with ID = 2, and the memory ME of three images with ID = 0 are used.

[0049] Thus, in this embodiment, if the target frame image FM-T is the Mth frame image FM-M, the determination unit 113 determines the object region RO in the Mth frame image FM-M based on the similarity between the Mth frame image FM-M and the 2nd to M-1th frame images FM-2 to FM-(M-1). More specifically, the determination unit 113 selects L (L is an integer between 1 and M-1) frame images FM in descending order of similarity between each of the 2nd to M-1th frame images FM-2 to FM-(M-1) and the Mth frame image FM-M. Then, the determination unit 113 determines the object region RO in the Mth frame image FM-M based on the memory ME (similarity memory ME-R) of the L frame images FM. In this embodiment, in addition to the similarity memory ME-R, a prompt specification memory ME-P is also used. Therefore, it can also be said that the determination unit 113 determines the object region RO in the Mth frame image FM-M based on the memory ME of L frame images FM (similar memory ME-R) and the memory ME of the prompt-specified frame image FM-P (prompt-specified memory ME-P). The six similar frame images FM-R are examples of L (where L is an integer between 1 and M-1) frame images FM. They are also examples of the first frame image FM-1.

[0050] By using similar memory ME-R to determine the mask data MD of the target frame image FM-T, the accuracy of the mask data MD can be improved compared to the comparative example described later (where memory ME of the frame image FM immediately preceding the target frame image FM-T is used).

[0051] Furthermore, an importance level may be assigned to each similar memory ME-R according to its similarity. For example, in this embodiment, six similar memory ME-Rs are selected. The six similar memory ME-Rs can be ranked from 1st to 6th in order of their similarity. In this case, the similar memory ME-Rs with higher rankings may be assigned a higher importance level.

[0052] The method for setting importance is arbitrary, but for example, a coefficient can be applied to the numerical data memory ME (similar memory ME-R) such that the lower the similarity rank, the smaller the coefficient. The coefficient could be, for example, 0.9 Ｓ (S can be set to "rank - 1"). In this case, the similar memory ME-R with the 1st similarity rank will have a coefficient of 1 (0.9 ０ ) However, the similar memory ME-R, which is ranked second in similarity, has a coefficient of 0.9 (0.9 １ ) However, the similar memory ME-R, which is ranked 3rd in similarity, has a coefficient of 0.81 (0.9 ２ Each of these values ​​is multiplied by the following factors. As a result, the values ​​of similar memory ME-R with lower ranks are relatively attenuated, and the values ​​of similar memory ME-R with higher ranks are relatively increased, thus highlighting the numerical data of memory ME with high similarity.

[0053] Thus, the determination unit 113 may change the importance of the six similar memory ME-R in determining the object region RO in the target frame image FM-T based on the degree of similarity between each of the six similar frame images FM-R and the target frame image FM-T. The six similar frame images FM-R are examples of L frame images FM. The target frame image FM-T is an example of the Mth frame image FM-M.

[0054] Figure 10 is a schematic diagram showing the difference in the method of selecting memory ME between the embodiment and the comparative example. Figure 10 shows 10 frame images FM-2 to FM-11. The target frame image FM-T is frame image FM-11. The target object OT is a circular object.

[0055] In the comparative example, the mask data MD-T of the target frame image FM-T is determined using memories ME-5 to ME-10 (indicated as "previous memory ME-B" in the figure) corresponding to the six frame images FM-5 to FM-10 immediately preceding the target frame image FM-T, frame image FM-11. However, for example, frame images FM-8 and FM-9 are black images, and frame image FM-7 is an image containing a non-circular object. When determining the mask data MD-T using memory ME based on such frame images FM, the accuracy of the mask data MD-T may be low.

[0056] On the other hand, in this embodiment, the mask data MD-T of the target frame image FM-T is determined using memories ME-2 to ME-6 and ME-10 (similar memory ME-R) corresponding to frame images FM-2 to FM-6 and FM-10 which are similar to the target frame image FM-T, which is frame image FM-11. Compared to the comparative example, by using memory ME of frame image FM which is closer to the target frame image FM-T,

[0057] Returning to the explanation of Figure 1, the display control unit 114 controls the display device 101 to display a region display image IR (IR-T) that shows the object region RO in the target frame image FM-T. By continuously displaying the region display images IR corresponding to all frame images FM of the video to be analyzed DG, a video (region display video) is displayed in which the object region RO in the video to be analyzed DG is continuously shown.

[0058] The display control unit 114 displays an image obtained from, for example, the target frame image FM-T and the mask data MD-T on the display device 101 as a region display image IR-T indicating the object region RO in the target frame image FM-T. More specifically, the display control unit 114 superimposes a mask image IM-T generated using, for example, the mask data MD-T onto the target frame image FM-T and displays it on the display device 101. At this time, instead of a black and white mask image IM as shown in Figure 5, for example, a mask image IM may be used in which an arbitrary color is displayed in the object region RO and the area outside the object region RO is colorless. Furthermore, pixels of the arbitrary color may be transparent. This makes it possible to display the state of the target object OT in the target frame image FM-T in an easy-to-understand manner.

[0059] Figure 9 is a schematic diagram showing an example of a region display image IR. Figure 9 shows a target frame image FM-T, a mask image IM-T corresponding to the target frame image FM-T, and a region display image IR-T obtained by superimposing the mask image IM-T onto the target frame image FM-T. Note that the mask image IM-T superimposed on the region display image IR-T is not a grayscale image, but an image in which transparent colors are displayed in the pixels corresponding to the target object OT. In the region display image IR-T, the region of the target object OT is shown in a single color, so the user can easily grasp the position and shape of the target object OT.

[0060] In this embodiment, the region display image IR is displayed on the display device 101 of the image processing device 10. However, for example, the region display image IR may be transmitted from the image processing device 10 to another information processing device via a network, and the region display image IR may be displayed on the display device 101 of the other information processing device.

[0061] [Flowchart] Figure 11 is a flowchart showing the operation of the processing unit 106. The processing unit 106 functions as an acquisition unit 111 and acquires image data of the video to be analyzed DG (step S101). As described above, the video to be analyzed DG contains N frame images FM-1 to FM-N. The N frame images FM-1 to FM-N are numbered (assigned IDs) from 1 to N in the order in which the N frame images FM-1 to FM-N are arranged in the video.

[0062] The processing unit 106 functions as a determination unit 113 and calculates the similarity between frame images FM (step S102). At this time, the determination unit 113 calculates the similarity between the frame image FM-K to be calculated and the frame images FM with an ID smaller than that of the frame image FM-K to be calculated. More specifically, if the frame image FM-K to be calculated is frame image FM-5 (ID is 5), then four similarities are calculated between frame image FM-5 and frame images FM-1 to FM-4. ​​Furthermore, in step S104, which will be described later, if the frame image FM that will become the prompt-specified frame image FM-P is known, the frame image FM that will become the prompt-specified frame image FM-P may be excluded from the similarity calculation.

[0063] The processing unit 106 functions as a determination unit 113 and generates a similarity list by associating the IDs of the top six similar frame images FM-R with the ID of the frame image FM-K with the ID of the frame image FM-K, and stores it in the storage device 105 (step S103). The IDs of the similar frame images FM-R in the similarity list are arranged in descending order of similarity. The determination unit 113 sequentially sets N frame images FM-1 to FM-N as the frame image FM-K and executes steps S102 and S103, generating a similarity list for all N frame images FM-1 to FM-N, and stores the similarity list in the storage device 105.

[0064] Note that the processing in steps S102 and S103 may be performed, for example, between steps S107 and S108. In this case, the target frame image FM-T is set as the frame image FM-K to be calculated, and a similar frame image FM-R may be selected for each frame image FM.

[0065] Next, the processing unit 106 functions as a receiving unit 112 and receives a prompt PT specifying the target object OT for the first frame image FM-1 (step S104). Thereafter, the first frame image FM-1 for which prompt PT has been specified becomes the prompt-specified frame image FM-P.

[0066] The processing unit 106 functions as a determination unit 113 and determines the mask data MD-P of the prompt-specified frame image FM-P (step S105). The processing unit 106 also functions as a determination unit 113 and generates a prompt-specified memory ME-P using the prompt-specified frame image FM-P and the mask data MD-P (step S106). The generated prompt-specified memory ME-P is stored in the memory bank 26.

[0067] Next, the processing unit 106 functions as a determination unit 113 and designates the target frame image FM-T as frame image FM with ID = 2, i.e., frame image FM-2 (step S107). The processing unit 106 functions as a determination unit 113 and refers to the similarity list of frame image FM-2 and reads out the memory ME corresponding to the ID listed in the similarity list, i.e., six similar memory ME-R from the memory bank 26 (step S108).

[0068] The processing unit 106 functions as a determination unit 113 and uses the target frame image FM-T (frame image FM-2), six similar memories ME-R, and prompt specification memory ME-P to determine the mask data MD-T of the target frame image FM-T (step S109). The processing unit 106 also functions as a determination unit 113 and uses the target frame image FM-T and the mask data MD-T to generate memory ME-T (step S110). Memory ME-T is stored in memory bank 26.

[0069] The processing unit 106 continues until the ID of the target frame image FM-T becomes N, that is, until the target frame image FM-T becomes frame image FM-N (step S111: NO), by setting the frame image FM with an ID obtained by adding 1 to the ID of the current target frame image FM-T as the target frame image FM-T (step S112), and returning to step S108. For example, if the current target frame image FM-T is frame image FM-2, then frame image FM-3 becomes the next target frame image FM-T.

[0070] Furthermore, when the ID of the target frame image FM-T becomes N, that is, when the target frame image FM-T becomes frame image FM-N (step S111: YES), the processing unit 106 functions as a display control unit 114 and displays the region display video on the display device 101 (step S113). The region display video is a continuous display of region display images IR. The display control unit 114 may automatically start displaying the region display video on the display device 101 after completing a predetermined process (for example, the process in step S110 for frame image FM-N). After that, the processing unit 106 terminates the process of this flowchart.

[0071] [Summary of Embodiments] As described above, the image processing apparatus 10 according to the embodiment utilizes a similar memory ME-R, which is the memory ME of a similar frame image FM-R of the target frame image FM-T, when estimating the object region RO in the target frame image FM-T. This makes it possible to accurately estimate the object region RO even when, for example, an image with little relation to the target frame image FM-T is displayed immediately before the target frame image FM-T, or when the target object OT disappears from the screen immediately before the target frame image FM-T.

[0072] The image processing device 10 according to this embodiment does not treat multiple similar memory ME-Rs as the same, but rather sets their importance based on the ranking of their similarity. This allows important (highly similar) similar memory ME-Rs to be emphasized and used for estimating the object region RO, thereby further improving the accuracy of the object region RO estimation.

[0073] In contrast, in the comparative example, when estimating the object region RO in the target frame image FM-T, the memory ME of the frame image FM immediately preceding the target frame image FM-T, which is the previous memory ME-B, is used. Therefore, for example, if an image unrelated to the target frame image FM-T is displayed immediately before the target frame image FM-T, or if the target object OT disappears from the screen immediately before the target frame image FM-T, the accuracy of estimating the object region RO will be significantly reduced.

[0074] The image processing apparatus 10 according to this embodiment can solve the problems of the comparative example and improve the accuracy of semantic segmentation of an image.

[0075] [Other] (1) In the embodiments described above, ROM and RAM were given as examples of the storage device 105, but the storage device 105 may be a flexible disk, magneto-optical disk (e.g., compact disk, digital multipurpose disk, Blu-ray® disk), smart card, flash memory device (e.g., card, stick, key drive), CD-ROM (Compact Disc-ROM), register, removable disk, hard disk, floppy® disk, magnetic strip, database, server, or other suitable storage medium.

[0076] (2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0077] (3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0078] (4) In the embodiments described above, the determination may be made by a value represented by one bit (0 or 1), by a Boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0079] (5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be rearranged in order, as long as there is no inconsistency. For example, in the methods described herein, various step elements are presented using an exemplary order and are not limited to the specific order presented.

[0080] (6) Each function illustrated in Figure 1 is implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each functional block is not particularly limited. That is, each functional block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired, wireless, etc.). A functional block may also be implemented by combining the above one device or the above multiple devices with software.

[0081] (7) The programs illustrated in the embodiments described above should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages ​​or by any other name.

[0082] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0083] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

[0084] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0085] (10) In the embodiments described above, the portable device may be a Mobile Station (MS). A Mobile Station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “mobile station,” “user terminal,” “user equipment (UE),” and “terminal” may be used interchangeably.

[0086] (11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0087] (12) In the embodiments described above, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on".

[0088] (13) The terms “determinating” and “deciding” as used in this disclosure may encompass a wide variety of actions. “Determinating” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (for example, searching in a table, database or another data structure), and confirming. Furthermore, "judgment" and "decision" may include considering something as a "judgment" or "decision" based on actions such as receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and access (e.g., accessing data in memory). Additionally, "judgment" and "decision" may include considering something as a "judgment" or "decision" based on actions such as resolving, selecting, choosing, establishing, and comparing. In short, "judgment" and "decision" may include considering something as a "judgment" or "decision" based on some action. Furthermore, "judgment (decision)" may be reinterpreted as "assuming," "expecting," or "considering."

[0089] (14) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0090] (15) Where articles are added in translation, for example, a, an, and the in English, the disclosure may include the fact that the noun following these articles is plural.

[0091] (16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0092] (17) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of certain information (e.g., notification that "it is X") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0093] 10...Image processing device, 101...Display device, 102...Input device, 103...Communication device, 104...Interface, 105...Storage device, 106...Processing device, 111...Acquisition unit, 112...Reception unit, 113...Decision unit, 114...Display control unit, DG...Video to be analyzed, FM (FM-1 to FM-N)...Frame image, FM-P...Prompt-specified frame image, FM-R...Similar frame image, FM-T...Target frame image, IM (IM-P, IM-T)...Mask image, IR (IR-T)...Region display image, MD (MD-1 to MD-N, MD-P, MD-T)...Mask data, ME (ME-1 to ME-N)...Memory, ME-B...Previous memory, ME-P...Prompt-specified memory, ME-R...Similar memory, RO...Object region, SM...Segmentation model.

Claims

1. An image processing apparatus comprising: an acquisition unit that acquires a plurality of frame images including a target frame image; a determination unit that determines the region of a first object in the target frame image based on the similarity between the target frame image and at least one frame image included in the plurality of frame images; and a display control unit that controls a display device to display an image showing the region of the first object in the target frame image.

2. The plurality of frame images are N (N is an integer of 2 or more) frame images that constitute a video, the N frame images are numbered from 1 to N in the order in which the N frame images are arranged in the video, the target frame image is the M (M is an integer of 2 or more and less than or equal to N)th frame image, and the determination unit determines the region of the first object in the Mth frame image based on the similarity between the Mth frame image and the 2 to M-1th frame images, the image processing apparatus according to claim 1.

3. The image processing apparatus according to claim 2, wherein the determination unit determines the region of the first object in each of the N frame images in the order of the numbered N frame images.

4. The image processing apparatus according to claim 2, wherein the determination unit selects L (where L is an integer between 1 and M-1) frame images in descending order of similarity between each of the 2 to M-1 frame images and the M frame image, and determines the region of the first object in the M frame image based on the feature data of the L frame images.

5. The image processing apparatus according to claim 4, wherein the determination unit changes the importance of the feature data of each of the L frame images in determining the region of the first object in the M frame image based on the degree of similarity between each of the L frame images and the M frame image.

6. The image processing apparatus according to claim 2, further comprising a receiving unit for receiving the designation of the first object for the first frame image among the N frame images.

7. The determination unit selects L (where L is an integer between 1 and M-1) frame images in descending order of similarity between each of the 2 to M-1 frame images and the M frame image, and determines the region of the first object in the M frame image based on the feature data of the L frame images and the feature data of the first frame image, as described in claim 6.

8. The image processing apparatus according to claim 1, wherein the determination unit determines mask data indicating the region of the first object in the target frame image, and the display control unit causes the display device to display the image obtained from the target frame image and the mask data as an image indicating the region of the first object in the target frame image.

9. A display control method comprising: acquiring a plurality of frame images including a target frame image; determining the region of a first object in the target frame image based on the similarity between the target frame image and at least one frame image included in the plurality of frame images; and controlling a display device to display an image showing the region of the first object in the target frame image.

Images