Inspection system, inspection method, and program

Abstract

An inspection system (100) that is for inspecting the accuracy of a motion vector and includes at least one processor. On the basis of the motion vector, the at least one processor applies, to an n-th input frame, motion compensation relating to a motion in a reverse direction from the n-th input frame to an (n-1)-th input frame, thereby acquiring an (n-1)-th reverse motion compensation input frame generated by respectively writing the pixel values of pixels of the n-th input frame to the pixels corresponding to the motion in the reverse direction. The at least one processor calculates the differences between the pixel values of pixels in the (n-1)-th input frame and the pixel values of pixels in the (n-1)-th reverse motion compensation input frame, thereby acquiring difference information relating to the accuracy of the motion vector.

Description

Testing systems, testing methods, programs

[0001] This invention relates to an inspection system, an inspection method, and a program.

[0002] Conventionally, a technique (super-resolution) has been known that uses machine learning models to estimate high-resolution still images based on low-resolution still images (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of this application are considering applying the above technology to moving images such as games, and estimating higher-quality moving images by considering not only the information of the current frame but also the information of past frames. In moving images, there is movement from past frames to the current frame, and if the information of past frames is directly input into a machine learning model, a ghosting phenomenon may occur in the estimated image output by the machine learning model, where afterimages are displayed. To prevent the ghosting phenomenon, it is advisable to apply motion compensation using motion vectors that show the movement from past frames to the current frame.

[0005] However, due to various factors, inaccurate motion vectors may be applied. If the motion vector contains incorrect information, the accuracy of the estimation results will decrease, and the ghosting phenomenon described above will occur. In this case, it is best to reset the motion vector to correct information, but it is not easy to verify whether the motion vector contains correct information or not.

[0006] This disclosure aims to provide an inspection system, inspection method, and inspection program that can inspect the accuracy of motion vectors in an image processing method that estimates high-quality moving images using information from past frames.

[0007] The inspection system according to this disclosure is an image processing system in which an nth estimated frame is output from a machine learning model when an nth (where n is a natural number of 2 or more) input frame, a motion vector indicating movement from the n-th input frame to the nth input frame, and n-th auxiliary information based on n-th feature information indicating the features of the n-th input frame are input to the machine learning model, and the inspection system includes at least one processor for inspecting the accuracy of the motion vector, wherein the at least one processor obtains an n-th inverse motion compensated input frame by applying motion compensation for the inverse movement from the nth input frame to the n-th input frame based on the motion vector, and writing the pixel value of each pixel in the nth input frame to each pixel corresponding to the inverse movement, and obtains difference information regarding the accuracy of the motion vector by calculating the difference between the pixel value of each pixel in the n-th input frame and the pixel value of each pixel in the n-th inverse motion compensated input frame.

[0008] This figure shows an example of the hardware configuration of the image processing system and the inspection system. This figure schematically shows the processing of the image processing system. This figure schematically shows the processing of the image processing system. This is a functional block diagram showing an example of the functions realized by the image processing system. This figure explains the processing of the rendering unit. This figure explains the processing in the input frame acquisition unit. This figure shows an overview of the generation of difference frames by applying reverse motion compensation. This is a functional block diagram showing an example of the functions realized by the inspection system. This is a flowchart showing an example of the processing flow executed by the inspection system.

[0009] Hereinafter, an example of an embodiment of the image processing system 1 and inspection system 100 according to the present invention will be described with reference to the drawings.

[0010] [1. Hardware Configuration of the Image Processing System] Figure 1 shows an example of the hardware configuration of the image processing system 1. The image processing system 1 is a computer such as a game console (game machine).

[0011] As shown in Figure 1, the image processing system 1 includes a control unit 10 containing at least one processor, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.

[0012] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1, for example. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images to a frame buffer based on graphics commands and data supplied from the CPU.

[0013] The memory unit 12 includes, for example, a main memory such as ROM or RAM and an auxiliary memory such as an HDD or SSD. The memory unit 12 stores instructions executed by the processor and the machine learning model 200. Specifically, the memory unit 12 stores programs executed by the control unit 10 and the trained machine learning model 200. The machine learning model 200 is trained using training data that includes training input videos and training output videos. The memory unit 12 has an area reserved for a frame buffer on which images are rendered by the GPU. In addition to programs for realizing various functions of the image processing system 1 described later, the memory unit 12 may also store, for example, a game program (game software).

[0014] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0015] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives user input and outputs a signal indicating the content of that input to the control unit 10.

[0016] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to the instructions of the control unit 10.

[0017] The audio output unit 19 is, for example, a speaker, and outputs the audio indicated by the audio data generated by the image processing system 1.

[0018] In addition to the devices described above, the image processing system 1 may also include an optical disc drive for reading optical discs such as DVD-ROMs and Blu-ray® discs, and a USB (Universal Serial Bus) port.

[0019] [2. Overview of the Image Processing System] Figures 2 and 3 are schematic diagrams illustrating the processing of the image processing system 1. In this embodiment, an example is given in which the image processing system 1 is used to improve the image quality of gameplay videos in a game. Gameplay videos are videos generated in response to the game program executed by the control unit 10 and user input received by the operation unit 16, and consist of multiple still images (frames) which are time-series data. The processing performed by the image processing system 1 is mainly as follows.

[0020] (1) Generation of Processing Frames First, the image processing system 1 generates an image (processing frame) in which a game object is drawn by rendering three-dimensional data showing one or more game objects viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial number of pixels) and a predetermined image quality (initial image quality). Processing frames are generated at predetermined time intervals. The number of pixels in a processing frame is, for example, 1920 × 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 and used for subsequent processing. In the following explanation, the processing of the nth (2 ≤ n ≤ N, n and N are natural numbers greater than or equal to 2) processing frame 20_n will be mainly used as an example, but similar processing is performed on other processing frames as well (i.e., n = 2, 3, ..., N).

[0021] (2) Acquisition of input frame The image processing system 1 acquires a frame (input frame) 22_n having a number of pixels greater than the initial number of pixels (input number of pixels) based on the acquired processing target frame 20_n. The input number of pixels is, for example, 3840 x 2160 (4K). Specifically, the input frame 22_n is generated by performing scaling and interpolation processing on the processing target frame 20_n.

[0022] Here, it should be noted that although the input frame 22_n has more pixels than the processing frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame is not simply determined by the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, the high signal-to-noise ratio, the high fidelity of spatial frequencies, and the high temporal stability (the low level of artifacts and flickering when multiple frames are displayed consecutively), either individually or as a combination of these factors, when compared to a reference frame.

[0023] (3) Acquisition of Estimated Frame The image processing system 1 inputs the input frame 22_n to the machine learning model 200 and acquires the estimated frame 24_n. The estimated frame 24_n is an image with the same number of pixels as the input number of pixels (estimated number of pixels) and an image quality that is equal to or better than the initial image quality (estimated image quality). Here, in addition to the input frame 22_n, the machine learning model 200 receives the (n-1)th auxiliary information 28_n-1 (see Figures 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 which shows the features of the first to (n-1)th input frames 22.

[0024] (4) The machine learning model 200 for acquiring cumulative feature information has a cumulative feature information output layer 202 that receives input frames 22_n and auxiliary information 28_n-1 as inputs and outputs the nth cumulative feature information 26_n which represents the features of the first to nth input frames 22 (see Figure 2). The image processing system 1 acquires the nth cumulative feature information 26_n.

[0025] The acquired nth cumulative feature information 26_n is input to the estimation frame output layer 204, and the estimation frame output layer 204 outputs the nth estimated frame 24_n. The acquired nth cumulative feature information 26_n is also stored in the memory unit 12 and used to estimate the next processing target frame (the (n+1)th processing target frame) 20_n+1, which corresponds to the estimated frame 24_n+1.

[0026] (5) Acquisition of auxiliary information The n-1st cumulative feature information 26_n-1 is, as described above, information that shows the characteristics of the 1st to n-1st input frames 22 (and by extension, the 1st to n-1st processing target frames 20). If the cumulative feature information 26_n-1, which has accumulated information on past processing target frames 20 in this way, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, so a high-quality estimated frame 24_n can be obtained.

[0027] If there is movement or other action in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, and the nth input frame 22_n and the cumulative feature information 26_n-1 are directly input to the machine learning model 200, a phenomenon may occur where a ghost image of the game object displayed in the (n-1)th processing frame 20_n-1 is displayed.

[0028] Therefore, the image processing system 1 obtains the n-th auxiliary information 28_n-1 by applying various corrections, described later, to the cumulative feature information 26_n-1, based on information obtained during rendering (such as motion vectors and depth buffers) (see Figures 2 and 3). The obtained n-th auxiliary information 28_n-1 is input to the machine learning model 200 along with the nth input frame 22_n, as described above, and is used to estimate the nth estimated frame 24_n.

[0029] [3. Functions Realized by the Image Processing System] Figure 4 is a functional block diagram showing an example of the functions realized by the image processing system 1. As shown in Figure 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a machine learning model storage unit 412, an estimated frame acquisition unit 414, an auxiliary information acquisition unit 416, a motion vector acquisition unit 418, a depth information acquisition unit 420, and an occurrence pixel identification unit 422.

[0030] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, input frame acquisition unit 410, estimated frame acquisition unit 414, auxiliary information acquisition unit 416, motion vector acquisition unit 418, depth information acquisition unit 420, and occurrence pixel identification unit 422 are mainly implemented by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 412 are mainly implemented by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by the game software.

[0031] [Game Processing Unit] The game processing unit 400 performs various processes related to the game. For example, the game processing unit 400 performs processes such as placing game objects O in the virtual 3D space VS, operating or moving the game objects O, and changing the viewpoint C from which the virtual 3D space VS is viewed, in response to the game program executed by the control unit 10 or the user input received by the operation unit 16. The game objects O are composed of primitives such as polygons represented by 3D data. The 3D data includes geometric information that indicates the position of vertices, phase information that indicates how the vertices are connected, and attribute information such as color.

[0032] [Rendering Unit] Figure 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates the first to Nth (N is a natural number of 2 or more) frames 20 to be processed by rendering (drawing) three-dimensional data representing one or more game objects O as seen from a predetermined viewpoint C. The rendering unit 402 performs rendering based on the results of various processes performed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes coordinate transformation processing from the view coordinate system to the screen coordinate system (perspective projection), and the perspective projection matrix (camera matrix) used in the coordinate transformation processing has a numerical value related to the change in viewpoint C added to it, as will be described later. The rendering unit 402 may also perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.

[0033] Here, the rendering unit 402 generates each processing target frame 20 by performing rendering such that the viewpoint C changes with each processing target frame 20. Even if the game processing unit 400 fixes the viewpoint C in a predetermined position, the rendering unit 402 adds a change to the viewpoint C with each processing target frame 20. As a result, as shown in Figure 5, the position of the displayed game object O changes in each of the processing target frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing target frame 20. Specifically, the rendering unit 402 changes the viewpoint C with each processing target frame 20 by adding a numerical value corresponding to a size less than one pixel, which differs for each processing target frame 20, to the perspective projection matrix. The rendering unit 402 changes the viewpoint C with each processing target frame 20 according to a predetermined rule. For example, the Halton column can be used as such a rule.

[0034] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for rendering processing in the rendering unit 402 and information obtained as a result of rendering processing. For example, the rendering information storage unit 404 stores the frame 20 to be processed. The rendering information storage unit 404 also stores fluctuation information, motion vectors, and depth information. Details of fluctuation information, motion vectors, and depth information will be described later. In addition, the rendering information storage unit 404 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.

[0035] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to the Nth processing frame 20. Specifically, the processing frame acquisition unit 406 acquires the first to the Nth processing frame 20 stored in the rendering information storage unit 404.

[0036] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information. The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. Specifically, the variation information is information that shows the amount of change of viewpoint C between before and after the change. The information that shows the amount of change can also be said to be a variation vector that shows the direction and distance of the change. For example, since the aforementioned Halton column contains information that shows the amount of change of viewpoint C, that information may be used as variation information.

[0037] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to the Nth input frames 22 by generating an input frame 22 that corresponds to the processing target frame 20 and has an input pixel count equal to or greater than the initial pixel count, based on each processing target frame 20. In this embodiment, each input frame 22 has an input pixel count greater than the initial pixel count. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 corresponding to the input frame 22.

[0038] Specifically, the input frame acquisition unit 410 obtains pixel values at positions corresponding to each pixel before variation in the processing target frame 20 by interpolation based on the variation information and each pixel of each processing target frame 20, and generates each input frame 22. FIG. 6 is a diagram for explaining the processing in the input frame acquisition unit 410. In FIG. 6, the case of obtaining the n-th input frame 22_n is illustrated. For example, as shown in FIG. 6, if the pixel center of a certain pixel in the input frame 22_n to be obtained is P １，０ , １，０ then the input frame acquisition unit 410 determines the pixel center P １，０ in the processing target frame 20_n that is closest to P ０，０ and the pixel values of the four pixels closest to P １，０ and the pixel values of the four pixels closest to P ０，１ and the pixel values of the four pixels closest to P １，１ Based on the coordinates and pixel values of P １，０ the pixel value of P １，０ is obtained by bilinear interpolation. Here, P' １，０ is at a position shifted from P

[0039] When rendering is executed such that the viewpoint C varies for each processing target frame 20, the amount of information in the time series increases. If each processing target frame 20 obtained in this way (hereinafter referred to as "variation processing target frame") is used for estimation, a higher-quality estimated frame 24 can be obtained.

[0040] On the other hand, if the variation processing target frame (or an image obtained by enlarging this) is directly input to the machine learning model 200, there is a possibility that the accuracy of the estimation may decrease due to the influence of the variation of the viewpoint C as described above.

[0041] Therefore, in the image processing system 1, as described above, based on the variation information and each pixel of each processing target frame 20, the pixel value at the position corresponding to each pixel before the variation in the processing target frame 20 is obtained by interpolation, each input frame 22 is generated, and this is input to the machine learning model 200. As a result, the influence of the variation of the viewpoint C is corrected, so that it is possible to suppress the decrease in the accuracy of the estimation.

[0042] [Machine learning model storage unit] The machine learning model storage unit 412 stores the machine learning model 200. Specifically, the machine learning model storage unit 412 stores the parameters of the machine learning model 200 (the number of convolutional layers, the number of nodes used in each convolutional layer, the weights of each node, etc.).

[0043] [Machine learning model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n and the (n - 1)th auxiliary information 28_n - 1. Specifically, the machine learning model 200 is a convolutional neural network (CNN: Convolutional Neural Network). As the machine learning model 200, for example, a known model such as ResNet having a multi-layer structure with a residual connection mechanism or a so-called encoder-decoder type U-Net can be used. The model described in Non-Patent Document 1 may be used as the machine learning model 200.

[0044] The machine learning model 200 is a model learned by a plurality of training data each including a learning input frame having an input pixel number and a learning estimated frame having an estimated pixel number. For the learning of the machine learning model 200, various known methods such as the error backpropagation method can be used.

[0045] Specifically, the machine learning model 200 includes a cumulative feature information output layer 202, an estimated frame output layer 204, and a convolutional layer 206 (see FIG. 2).

[0046] The cumulative feature information output layer 202 receives the nth input frame 22_n and the nth-1st auxiliary information 28_n-1 based on the nth-1st cumulative feature information 26_n-1 that shows the features of the first to nth-1st input frames 22, and outputs the nth cumulative feature information 26_n that shows the features of the first to nth input frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (information in bitmap format) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be said to be a feature map that shows the features of the first to nth-1st input frames 22.

[0047] The cumulative feature information output layer 202 receives the first input frame 22_1 and the given auxiliary information, and outputs the first cumulative feature information 26_1. When n=1, since there is no previous cumulative feature information 26 and auxiliary information 28, the pre-prepared given auxiliary information is input to the cumulative feature information output layer 202 along with the first input frame 22_1.

[0048] The estimated frame output layer 204 receives the nth cumulative feature information 26_n as input and outputs the nth estimated frame 24_n. The estimated frame output layer 204 may be composed of, for example, one or more convolutional layers, similar to the cumulative feature information output layer 202. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (inverse convolutional layers).

[0049] The convolutional layer 206 is a layer that reduces the number of channels in the cumulative feature information 26 while maintaining the number of pixels. The cumulative feature information 26 output from the convolutional layer 206 is used for processing in the auxiliary information acquisition unit 416. The convolutional layer 206 reduces the dimension of the cumulative feature information 26, thus reducing computational costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1x1, but is not limited to this.

[0050] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 inputs each input frame 22 to the machine learning model 200 and acquires the first to nth estimated frames 24, each having an estimated number of pixels greater than or equal to the input number of pixels and greater than or equal to the initial number of pixels. In this embodiment, the estimated frame 24 has the same number of estimated pixels as the input number of pixels. More specifically, the estimated frame acquisition unit 414 inputs the nth input frame 22_n and the n-1 auxiliary information 28_n-1 to the machine learning model 200 and acquires the nth estimated frame 24_n.

[0051] [Motion Vector Acquisition Unit] The motion vector acquisition unit 418 acquires the n-th motion vector, which is information indicating the amount and direction of movement from the n-th processing target frame 20_n-1 to the nth processing target frame 20_n. A motion vector is also called a motion vector.

[0052] [Depth Information Acquisition Unit] The depth information acquisition unit 420 acquires the n-1st depth information, which indicates the depth of each pixel in the n-1st processing target frame 20_n-1, and the nth depth information, which indicates the depth of each pixel in the nth processing target frame 20_n. The depth information is also called a depth buffer, Z buffer, or depth buffer.

[0053] [Appearing Pixel Identification Unit] The appearing pixel identification unit 422 identifies the nth appearing pixel 222_n, which is a pixel in the nth input frame 22_n that displays all or part of the game object O that is not displayed in the nth input frame 22_n-1, based on the (n-1)th depth information and the nth depth information (see Figure 3). Specifically, the appearing pixel identification unit 422 identifies the nth appearing pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Alternatively, the appearing pixel identification unit 422 may identify the nth appearing pixel 222_n based on the (n-1)th perspective projection matrix related to the nth input frame 22_n-1 and the nth perspective projection matrix related to the nth input frame 22_n. Furthermore, the appearing pixel identification unit 422 may identify the nth appearing pixel 222_n using the (n-1)th motion vector. More specifically, the occurrence pixel identification unit 422 identifies the nth occurrence pixel and generates the nth occurrence pixel value, which is image information indicating the position of the nth occurrence pixel 222_n.

[0054] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 416 acquires the n-th auxiliary information 28_n-1 by applying motion compensation to the n-th cumulative feature information 26_n-1 based on the n-th motion vector. Motion compensation refers to the process of moving the pixel at position x in the n-th cumulative feature information 26_n to position x' when, for example, a pixel at position x in the n-th input frame 22_n-1 has moved to position x' in the nth input frame 22_n. That is, the auxiliary information acquisition unit 416 acquires the n-th auxiliary information 28_n-1 by setting the pixel value of one or more pixels in the n-th cumulative feature information 26_n-1 to the pixel at the position where the pixel has moved according to the amount and direction of the pixel's movement, based on the n-th motion vector.

[0055] If there is movement of the game object O between the nth processing target frame 20_n and the (n-1)th processing target frame 20_n-1, and the nth estimated frame 24_n is obtained by directly inputting the nth input frame 22_n and the (n-1)th cumulative feature information 26_n-1 into the machine learning model 200, a ghosting phenomenon may occur in the output nth estimated frame 24_n, where the afterimage of the game object O that was displayed in the nth input frame 22_n is displayed.

[0056] Therefore, in the image processing system 1, motion compensation is applied to the n-th cumulative feature information 26_n-1 based on the n-th motion vector, as described above, to obtain the n-th auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this n-th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghosting phenomenon described above.

[0057] [4. Inspection System] As shown in Figure 1, the inspection system 100 may have a hardware configuration similar to that of the image processing system 1. That is, the inspection system 100 may include a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19. Note that some of these units may be common with the units of the image processing system 1. Furthermore, the inspection system 100 is not limited to one computer, but may be composed of multiple computers.

[0058] Figure 7 is a diagram illustrating the overview of the processing in the inspection system. The inspection system 100 is a computer for inspecting the accuracy of motion vectors. Figure 7 shows an example of inspecting the accuracy of the (n-1)th motion vector, which represents the movement from the past frame, input frame 22_n-1, to the current frame, input frame 22_n.

[0059] Here, the motion vector may be incorrect due to various factors. If the motion vector is incorrect, the accuracy of the estimation results in the machine learning model 200 decreases, and the ghosting phenomenon occurs. In this case, it is best to reset the motion vector to the correct value. However, since the motion vector cannot be visually confirmed, it is not easy to verify its accuracy. Also, the cause of the ghosting phenomenon is not necessarily the motion vector; it may have occurred due to errors in other parameters.

[0060] Therefore, in this embodiment, motion compensation for reverse motion (hereinafter also referred to as reverse motion compensation) is applied to the current frame, input frame 22_n, to generate a reverse motion compensation input frame 29_n-1. The difference between the reverse motion compensation input frame 29_n-1 and the past frame, input frame 22_n-1, is calculated to obtain a difference frame related to the accuracy of the motion vector. The difference frame is then displayed on the display unit 18 to visually confirm the accuracy of the motion vector.

[0061] Figure 7 shows how object A, which is positioned in front of the background, moves to the right as it moves from input frame 22_n-1 to input frame 22_n. Therefore, the direction indicated by the (n-1)th motion vector for object A is to the right.

[0062] Here, the inverse motion vector used for inverse motion compensation is a vector with the same magnitude as the motion vector but pointing in the opposite direction. Therefore, the direction indicated by the (n-1)th inverse motion vector is to the left.

[0063] When the (n-1)th inverse motion compensation is applied to input frame 22_n, for example, the pixel value of pixel p1 in Figure 7 is written to pixel p1 in the inverse motion compensated input frame 29_n-1. This is because there is no movement in pixel p1 between input frame 22_n-1 and input frame 22_n.

[0064] On the other hand, for example, the pixel value of pixel p2 in Figure 7 is written to pixel p3 in the reverse motion compensation input frame 29_n-1. For example, if the pixel value of pixel p2 in input frame 22_n indicates blue, then blue will be displayed in pixel p3 in the reverse motion compensation input frame 29_n-1.

[0065] Furthermore, in the case of pixels where multiple pixel values ​​may be written, it is preferable that only the color information contained in the pixel value that includes information indicating that the distance from viewpoint C is short (information indicating that it is in front of the screen) be written. For example, if the design is such that the depth information decreases as the distance from viewpoint C decreases, it is preferable that only the color information contained in the pixel value with small depth information be written. For this reason, the pixel value of pixel p3 in input frame 22_n should not be written to pixel p3 in the reverse motion compensation input frame 29_n. This is because pixel p3 in input frame 22_n represents the background of object A, and its depth information is greater than the depth information of pixel p2, which represents object A in input frame 22_n. Furthermore, if the design is such that the depth information increases as the distance from viewpoint C decreases, it is preferable that only the color information contained in the pixel value with large depth information be written.

[0066] Furthermore, it would be beneficial if the design prioritized depth information over color information when comparing which of multiple pixel values ​​to write. This would allow for efficient writing of pixel values ​​in pixels where duplicate values ​​might be written.

[0067] By writing the pixel values ​​as described above to all pixels, the inverse motion compensation input frame 29_n-1 shown in Figure 7 is generated.

[0068] As shown in Figure 7, no pixel value is written to pixel p2 in the reverse motion compensation input frame 29_n-1. That is, pixel p2 in the reverse motion compensation input frame 29_n-1 does not contain color information and is perceived as black. In this way, the reverse motion compensation input frame 29_n-1 may include unwritten pixels in the region corresponding to the pixels of input frame 22_n where object A, which is not displayed in input frame 22_n-1, is displayed, in which no pixel value is written.

[0069] Next, the difference between input frame 22_n-1 and reverse motion compensation input frame 29_n-1 is calculated. Specifically, a difference frame is generated by subtracting the pixel value of each pixel in input frame 22_n-1 from the pixel value of each pixel in reverse motion compensation input frame 29_n-1. If there is no error in the (n-1)th motion vector, the difference frame will be generated as a frame that does not contain pixel values. If there is an error in the (n-1)th motion vector, the difference frame will contain pixels that include color information.

[0070] By employing the inspection system 100 of this embodiment, for example, a game developer can visually check the difference frames displayed on the display unit 18 to confirm the accuracy of the motion vectors.

[0071] [5. Functions Realized in the Inspection System] Figure 8 is a functional block diagram showing an example of the functions realized in the inspection system 100. As shown in Figure 8, the inspection system 100 realizes an input frame acquisition unit 101, a motion vector acquisition unit 102, a reverse motion compensation application unit 103, a difference calculation unit 104, and a display control unit 105. Each of these units is mainly realized by the control unit 10.

[0072] [Input Frame Acquisition Unit] The input frame acquisition unit 101 acquires the input frame 22 generated in the image processing system 1.

[0073] [Motion Vector Acquisition Unit] The motion vector acquisition unit 102 acquires motion vectors generated in the image processing system 1.

[0074] [Reverse Motion Compensation Application Unit] The reverse motion compensation application unit 103 obtains a reverse motion compensated input frame by applying reverse motion compensation to the input frame 22 based on the motion vector. Specifically, the reverse motion compensation application unit 103 obtains a reverse motion compensated input frame generated by writing the pixel value of each pixel in the input frame 22 to the position corresponding to the reverse motion vector.

[0075] [Difference Calculation Unit] The difference calculation unit 104 calculates the difference between the pixel value of each pixel in the past input frame and the pixel value of each pixel in the reverse motion compensation input frame, and obtains a difference frame.

[0076] [Display Control Unit] The display control unit 105 causes the difference frame to be displayed on the display unit 18.

[0077] [6. Processing performed by the inspection system] Figure 9 is a flowchart showing an example of the processing flow performed by the inspection system 100. The processing shown in Figure 9 is performed by the control unit 10 operating according to the program stored in the storage unit 12.

[0078] First, the control unit 10 acquires input frames 22_n-1 and 22_n generated in the image processing system 1 (S101). The control unit 10 also acquires the (n-1)th motion vector indicating the movement from input frame 22_n-1 to input frame 22_n (S102).

[0079] Next, the control unit 10 applies reverse motion compensation to the input frame 22_n (S103). Specifically, the control unit 10 generates a reverse motion compensated input frame 29_n-1 by writing the pixel value of each pixel in the input frame 22_n to a position corresponding to the (n-1)th reverse motion vector, which is generated based on the (n-1)th motion vector.

[0080] Next, the control unit 10 generates a difference frame based on the input frame 22_n-1 and the reverse motion compensation input frame 29_n-1 (S104). Specifically, the control unit 10 generates the difference frame by subtracting the pixel value of each pixel in the input frame 22_n-1 from the pixel value of each pixel in the reverse motion compensation input frame 29_n-1. Then, the control unit 10 displays the difference frame on the display unit 18 (S105).

[0081] Note that while Figure 9 only describes the case where the accuracy of the (n-1)th motion vector, which shows the movement from input frame 22_n-1 to input frame 22_n, is checked, the same processing shown in Figure 9 should also be performed on motion vectors that show movement between other input frames 22.

[0082] [7. Summary] According to the inspection system 100 of this embodiment described above, the accuracy of the motion vector can be easily confirmed by visual inspection. As a result, it is possible to determine whether the cause of the ghosting phenomenon, etc., in the estimated frame 24 is an error in the value of the motion vector or an error in the value of another parameter. If it is found that the value of the motion vector is incorrect based on the difference frame, the game developer, etc., should reset the motion vector to the correct value.

[0083] In this embodiment, an example of generating a visually perceptible difference frame in the display unit 18 has been described, but the difference between the input frame 22_n-1 and the reverse motion compensation input frame 29_n-1 is not limited to being represented by frames. For example, difference information regarding the difference between the input frame 22_n-1 and the reverse motion compensation input frame 29_n-1 may be generated based on those frames. The difference information should be recognizable based on human perception.

[0084] Furthermore, although this embodiment describes an example in which auxiliary information 28_n-1 is generated based on the n-1st cumulative feature information 26_n-1 which shows the features of the first to n-1st input frames 22, the embodiment is not limited to this, and the auxiliary information 28_n-1 may be generated based on feature information which shows the features of at least the n-1st input frame 22_n-1.

[0085] In this embodiment, the example given is when the number of input pixels is greater than the initial number of pixels, and the number of input pixels and the estimated number of pixels are the same. However, the number of input pixels and the initial number of pixels may be the same, and the number of estimated pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the frame 20 to be processed.

[0086] [8. Addendum] For example, the inspection system can also be configured as follows: (1) An inspection system for inspecting the accuracy of the motion vector in an image processing system in which an estimated nth frame output from a machine learning model is obtained by inputting an nth (where n is a natural number greater than or equal to 2) input frame, a motion vector indicating movement from the n-th input frame to the nth input frame, and n-th auxiliary information based on n-th feature information indicating the features of the n-th input frame, wherein the at least one processor obtains an n-th inverse motion compensated input frame by applying motion compensation for the inverse movement from the nth input frame to the n-th input frame based on the motion vector, thereby writing the pixel value of each pixel in the nth input frame to each pixel corresponding to the inverse movement, and obtains difference information regarding the accuracy of the motion vector by calculating the difference between the pixel value of each pixel in the n-th input frame and the pixel value of each pixel in the n-th inverse motion compensated input frame. (2) The inspection system according to (1), wherein the difference information is generated by subtracting the pixel value of each pixel in the n-1 input frame from the pixel value of each pixel in the n-1 reverse motion compensation input frame. (3) The inspection system according to (1) or (2), wherein the n-1 reverse motion compensation input frame includes unwritten pixels in an area corresponding to a pixel in the n input frame where an object not displayed in the n-1 input frame is displayed, and the pixel value has not been written to it. (4) The inspection system according to any one of (1) to (3), wherein the pixel value includes information about color, and the at least one processor causes the difference information to be displayed on a display unit.(5) The inspection system according to any one of (1) to (4), wherein the motion vector is information relating to the direction and amount of movement from the n-1 input frame to the n input frame, and at least one processor obtains an inverse motion vector that is in the opposite direction to the motion vector and shows the same amount of movement, and obtains the n-1 inverse motion compensated input frame by applying inverse motion compensation relating to the movement in the reverse direction from the n input frame to the n-1 input frame to the n input frame based on the inverse motion vector. (6) The inspection system according to any one of (1) to (5), wherein at least one processor writes color information included in the pixel value that includes information indicating that the pixel is closer to the viewpoint among the multiple pixel values ​​in the n-1 inverse motion compensated input frame where multiple pixel values ​​may be written to the same pixel.

Claims

1. An inspection system for inspecting the accuracy of the motion vector in an image processing system in which an nth estimated frame is output from a machine learning model by inputting an nth input frame (where n is a natural number greater than or equal to 2), a motion vector indicating movement from the nth-1th input frame to the nth input frame, and nth-1st auxiliary information based on nth-1st feature information indicating the features of the nth-1st input frame, wherein the inspection system includes at least one processor, the at least one processor obtains an nth-1 inverse motion-compensated input frame by applying motion compensation for the inverse movement from the nth input frame to the nth input frame based on the motion vector, thereby writing the pixel value of each pixel in the nth input frame to each pixel corresponding to the inverse movement, and obtaining difference information regarding the accuracy of the motion vector by calculating the difference between the pixel value of each pixel in the nth-1st input frame and the pixel value of each pixel in the nth-1 inverse motion-compensated input frame.

2. The inspection system according to claim 1, wherein the difference information is generated by subtracting the pixel value of each pixel in the n-1 input frame from the pixel value of each pixel in the n-1 reverse motion compensation input frame.

3. The inspection system according to claim 1 or 2, wherein the n-1 reverse motion compensation input frame includes unwritten pixels in a region corresponding to a pixel of the nth input frame in which an object not displayed in the n-1 input frame is displayed, and in which the pixel value has not been written.

4. The inspection system according to claim 1 or 2, wherein the pixel value includes information about color, and the at least one processor causes the display unit to display the difference information.

5. The inspection system according to claim 1 or 2, wherein the motion vector is information relating to the direction and amount of motion of movement from the n-1 input frame to the n input frame, and the at least one processor obtains an inverse motion vector based on the motion vector, which is in the opposite direction and has the same amount of motion as the motion vector, and obtains the n-1 inverse motion compensated input frame by applying inverse motion compensation relating to the reverse movement from the n input frame to the n-1 input frame to the n input frame based on the inverse motion vector.

6. The inspection system according to claim 1 or 2, wherein at least one processor writes color information included in a pixel value that includes information indicating that the pixel value is close in distance from the viewpoint to the n-1 reverse motion compensation input frame, in a pixel where multiple pixel values ​​may be written to the n-1 reverse motion compensation input frame.

7. An inspection method for inspecting the accuracy of a motion vector in an image processing system in which an estimated nth frame output from a machine learning model is obtained by inputting an nth (where n is a natural number greater than or equal to 2) input frame, a motion vector indicating movement from the (n-1)th input frame to the nth input frame, and an (n-1)th auxiliary information based on the (n-1)th feature information indicating the features of the (n-1)th input frame, the inspection method comprising: a processor obtaining an (n-1)th inverse motion-compensated input frame generated by applying motion compensation for the inverse movement from the nth input frame to the n-1st input frame based on the motion vector, thereby writing the pixel value of each pixel in the nth input frame to each pixel corresponding to the inverse movement; and obtaining difference information regarding the accuracy of the motion vector by calculating the difference between the pixel value of each pixel in the n-1st input frame and the pixel value of each pixel in the n-1st inverse motion-compensated input frame.

8. A program for checking the accuracy of the motion vector in an image processing system in which an estimated nth frame output from a machine learning model is obtained by inputting an nth (where n is a natural number greater than or equal to 2) input frame, a motion vector indicating movement from the (n-1)th input frame to the nth input frame, and (n-1)th auxiliary information based on the (n-1)th feature information indicating the features of the (n-1)th input frame into a machine learning model, the program comprising: means for obtaining an (n-1)th inverse motion compensated input frame generated by applying motion compensation for the inverse movement from the nth input frame to the (n-1)th input frame based on the motion vector, thereby writing the pixel value of each pixel in the nth input frame to each pixel corresponding to the inverse movement; and means for obtaining difference information regarding the accuracy of the motion vector by calculating the difference between the pixel value of each pixel in the (n-1)th input frame and the pixel value of each pixel in the (n-1)th inverse motion compensated input frame.

Images