Inspection system, inspection method, and program

Abstract

The present invention makes it possible to easily determine whether or not an unintended visual effect appears when a high-quality moving image is subjected to predetermined image processing. Provided is an inspection system for a game program, the inspection system comprising a processor, a storage unit, and a display unit. The processor: acquires an input moving image including a plurality of input frames each having a predetermined number of input pixels; performs first processing, which is predetermined image processing, on a first processing frame, which is one of the plurality of input frames, to acquire a first processing result; inputs the input moving image to a machine learning model to acquire an output moving image including a plurality of output frames each having an estimated number of pixels larger than the number of input pixels; and performs second processing corresponding to the first processing on a second processing frame, which is an output frame corresponding to the first processing frame, to acquire a second processing result. The display unit displays the first processing result and the second processing result in comparison with each other. The machine learning model has been trained with training data including an input moving image for learning and an output moving image for learning.

Description

Inspection System, Inspection Method, and Program

[0001] The present invention relates to an inspection system, an inspection method, and a program.

[0002] Conventionally, a technique (super-resolution) for estimating a high-quality still image based on a low-quality still image using a machine learning model has been known (see Non-Patent Document 1 below).

[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014

[0004] The inventors of the present application have considered applying the above super-resolution to a moving image such as a game, and estimating a higher-quality moving image by considering not only the information of each frame to be processed at this time but also the information of past frames. Depending on the game, various image processes may be performed on the image to generate an image with a predetermined visual effect in order to enhance the user's sense of presence. However, since the parameters used for the image process are set based on the image before the super-resolution process, when the image process is performed on the image after the super-resolution process, an image with an unintended visual effect may be generated.

[0005] An object of the present disclosure is to provide an inspection system, an inspection method, and an inspection program that can easily determine whether an unintended visual effect appears when a predetermined image process is performed on a high-quality moving image in an image processing method for estimating a high-quality moving image based on a low-quality moving image using information of past frames.

[0006] A game program testing system comprising a processor, a storage unit for storing instructions executed by the processor and a machine learning model, and a display unit, wherein at least one processor acquires an input video including a plurality of input frames having a predetermined number of input pixels, applies a first processing which is a predetermined image processing to a first processing frame which is one of the plurality of input frames, and obtains a first processing result, inputs the input video to the machine learning model, acquires an output video including a plurality of output frames having an estimated number of pixels greater than the number of input pixels, applies a second processing which corresponds to the first processing to a second processing frame which is the output frame corresponding to the first processing frame, and obtains a second processing result, the display unit compares and displays the first processing result and the second processing result, and the machine learning model is trained using training data including training input video and training output video.

[0007] This is a diagram showing an example of the hardware configuration of an image processing system. This is a diagram showing an overview of the image processing system. This is a diagram schematically showing the processing of the image processing system. This is a functional block diagram showing an example of the functions realized by the image processing system. This is a diagram explaining the processing of the rendering unit. This is a diagram explaining the processing in the intermediate frame acquisition unit. This is a diagram comparing the first processing result and the second processing result related to the bloom effect. This is a diagram comparing the first processing result and the second processing result related to the histogram. This is a diagram comparing the first processing result and the second processing result related to the histogram. This is a diagram comparing the first processing result and the second processing result related to the heatmap. This is a flowchart showing an example of the processing flow executed by the image processing system. This is a flowchart showing an example of the processing flow executed by the image processing system.

[0008] Hereinafter, an example of an embodiment of the inspection system 1 according to the present invention will be described with reference to the drawings.

[0009] [1. Hardware Configuration of the Testing System] Figure 1 shows an example of the hardware configuration of testing system 1. Testing system 1 is a computer used, for example, in game development. As shown in Figure 1, testing system 1 includes a control device 10, a storage device 12, a communication device 14, an operating device 16, a display device 18, and an audio output device 19.

[0010] The control device 10 includes a program control device such as a CPU that operates according to a program installed in the inspection system 1, for example. The control device 10 also includes a GPU (Graphics Processing Unit) that draws images to a frame buffer based on graphics commands and data supplied from the CPU.

[0011] The storage device 12 includes, for example, main memory such as ROM or RAM, and auxiliary storage such as HDD or SSD. The storage device 12 stores instructions executed by the processor and the machine learning model 200. Specifically, the storage device 12 stores programs executed by the control device 10 and the trained machine learning model 200. Here, the machine learning model 200 is trained using training data including input video images and output video images for training, but the details will be described later. In addition, the storage device 12 has an area reserved for a frame buffer on which images are drawn by the GPU. Furthermore, in addition to programs for realizing the various functions of the inspection system 1 described later, the storage device 12 may also store, for example, a game program (game software).

[0012] The communication device 14 is, for example, a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0013] The operating device 16 is a user interface such as a keyboard, mouse, or game console controller, and it receives user input and outputs a signal indicating the content of that input to the control device 10.

[0014] The display device 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to the instructions of the control device 10. The display device 18 also compares and displays the first processing result and the second processing result, which will be described later.

[0015] The audio output device 19 is, for example, a speaker, and outputs the audio indicated by the audio data generated by the inspection system 1.

[0016] In addition to the devices described above, the inspection system 1 may also include an optical disc drive for reading optical discs such as DVD-ROMs and Blu-ray® discs, and a USB (Universal Serial Bus) port.

[0017] [2. Overview of the Inspection System] Figures 2 and 3 schematically show the processing performed by the inspection system 1. As shown in Figure 2, the machine learning model 200 includes a cumulative feature information output layer 202, an estimated frame output layer 204, a convolutional layer 206, an intermediate frame acquisition unit 310, and an auxiliary information acquisition unit 322. In this embodiment, an example is described in which the inspection system 1 improves image quality using the 1st to nth input frames 20, but the inspection system 1 may improve image quality by other means if it can improve image quality using past input frames 20. This disclosure is used to improve the image quality of gameplay videos in games. Gameplay videos are input videos that are input to the inspection system 1. Input videos are videos generated in response to a game development program executed by the control device 10 or user input received by the operation device 16, and consist of a plurality of still images (frames) which are time-series data. The inspection system 1 inputs the input videos to the machine learning model 200 and acquires output videos that include a plurality of output frames having an estimated number of pixels greater than the number of input pixels. The processes performed by the inspection system 1 are mainly as follows: (1) Generation of input frames

[0018] First, the inspection system 1 acquires an input video containing multiple input frames 20 having a predetermined number of input pixels. Specifically, the inspection system 1 generates an image (input frame 20) in which a game object is drawn by rendering three-dimensional data showing one or more game objects viewed from a predetermined viewpoint. This input frame 20 is an image having a predetermined number of pixels (input number of pixels) and a predetermined image quality (input image quality) (see Figure 3). An input frame 20 is generated at predetermined time intervals. The number of pixels in an input frame 20 is, for example, 1920 × 1080 (1080p). The input video consists of multiple input frames 20 arranged in chronological order. The generated input video is stored in the storage device 12 and used for subsequent processing. In the following explanation, the processing targeting the nth (nth) input frame 20_n will be mainly illustrated, but similar processing is performed for other input frames 20 as well (i.e., n = 2, 3, ..., N). (2) Acquisition of intermediate frames

[0019] The inspection system 1 inputs the input frame 20_n to the intermediate frame acquisition unit 310. Based on the acquired input frame 20_n, the intermediate frame acquisition unit 310 acquires a frame (intermediate frame 22_n) having a greater number of pixels (intermediate pixel count) than the input pixel count. The intermediate pixel count is, for example, 3840 x 2160 (4K). Specifically, the intermediate frame 22_n is generated by performing scaling and interpolation processing on the input frame 20_n (see Figure 3).

[0020] Here, it should be noted that although the intermediate frame 22_n has a greater number of pixels than the input frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame is not simply determined by the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, the high signal-to-noise ratio, the high reproducibility of spatial frequencies, and the high temporal stability (the low level of artifacts and flickering when multiple frames are displayed in succession) when compared to a reference frame, either individually or in combination. (3) Acquisition of cumulative feature information

[0021] The cumulative feature information output layer 202 receives the nth intermediate frame 22_n and the nth-1st auxiliary information 28_n-1 based on the nth-1st cumulative feature information 26_n-1 that shows the features of the first to nth intermediate frames 22, and outputs the nth cumulative feature information 26_n that shows the features of the first to nth intermediate frames 22_n.

[0022] The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (information in bitmap format) having the same number of pixels as the number of intermediate pixels. The cumulative feature information 26_n-1 can also be described as a feature map that shows the features of the first to the (n-1)th intermediate frames 22.

[0023] The cumulative feature information output layer 202 receives the first intermediate frame 22_1 and the given auxiliary information 28 as inputs and outputs the first cumulative feature information 26_1. When n=1, since there is no previous cumulative feature information 26 and auxiliary information 28, the pre-prepared given auxiliary information 28 is input to the cumulative feature information output layer 202 along with the first intermediate frame 22_1. As a result, the inspection system 1 obtains the nth cumulative feature information 26_n based on the intermediate frame 22_n and the auxiliary information 28_n-1. The obtained nth cumulative feature information 26_n is also stored in the memory device 12 and used to estimate the estimated frame 24_n+1 corresponding to the next input frame (the (n+1)th input frame) 20_n+1. (4) Acquisition of the estimated frame

[0024] The acquired nth cumulative feature information 26_n is input to the estimated frame output layer 204. The estimated frame output layer 204 acquires an estimated frame 24_n based on the cumulative feature information 26_n. The estimated frame 24_n is an image with the same number of pixels as the number of intermediate pixels (estimated number of pixels) and an image quality equal to or better than the input image quality (estimated image quality) (see Figure 3).

[0025] The estimated frame output layer 204 may be composed of, for example, one or more convolutional layers 206, similar to the cumulative feature information output layer 202. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (inverse convolutional layers). Since the estimated frame 24 is a frame output from the machine learning model, it will also be referred to as an output frame below. (5) Acquisition of auxiliary information

[0026] The acquired nth cumulative feature information 26_n is input to the convolutional layer 206. The convolutional layer 206 is a layer that reduces the number of channels in the cumulative feature information 26 while maintaining the number of pixels. The cumulative feature information 26 output from the convolutional layer 206 is used for processing in the auxiliary information acquisition unit 322. The convolutional layer 206 reduces the dimension of the cumulative feature information 26, thus reducing computational costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1x1, but is not limited to this.

[0027] As mentioned above, the n-1st cumulative feature information 26_n-1 represents the characteristics of the first to n-1st intermediate frames 22 (and consequently, the first to n-1st input frames 20). By using this cumulative feature information 26_n-1, which has accumulated information from past input frames 20, to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, thus enabling the acquisition of a high-quality estimated frame 24_n.

[0028] However, if there is movement or other action in the displayed game object between the (n-1)th input frame 20_n-1 and the nth input frame 20_n, and the nth intermediate frame 22_n and the cumulative feature information 26_n-1 are input directly to the cumulative feature information output layer 202, a phenomenon may occur where a ghost image of the game object displayed in the (n-1)th input frame 20_n-1 is displayed.

[0029] Therefore, the inspection system 1 applies various corrections, described later, to the cumulative feature information 26_n-1 based on information obtained during rendering (such as motion vectors and depth buffers) to acquire the (n-1)th auxiliary information 28_n-1 (see Figures 2 and 3). The acquired (n-1)th auxiliary information 28_n-1 is input to the cumulative feature information output layer 202 along with the nth intermediate frame 22_n, as described above, and is used to estimate the nth estimated frame 24_n. (6) Acquisition of the first processing result

[0030] The inspection system 1 inputs the input frame 20_n not only to the intermediate frame acquisition unit 310 but also to the first processing result acquisition unit 324. The first processing result acquisition unit 324 applies a predetermined image processing, which is the first processing (described later), to the input frame 20_n and acquires the first processing result. Hereinafter, the input frame 20 input to the first processing result acquisition unit 324 will be referred to as the first processing frame. (7) Acquisition of the second processing result

[0031] The inspection system 1 inputs the estimated frame 24_n to the second processing result acquisition unit 326. The second processing result acquisition unit 326 applies the second processing (described later) corresponding to the first processing frame (in this case, the input frame 20_n) to the output frame (in this case, the estimated frame 24_n), and acquires the second processing result. Hereinafter, the estimated frame 24 input to the second processing result acquisition unit 326 will be referred to as the second processing frame.

[0032] As described above, according to the inspection system 1 of this embodiment, the estimated frame 24 is estimated using auxiliary information 28, which contains past information, in addition to the intermediate frame 22 corresponding to the current input frame 20. This increases the amount of information available for estimation, so a high-quality estimated frame 24_n can be obtained. Furthermore, according to the inspection system 1 of this embodiment, a first processing result is obtained, which is the result of applying a first processing to the input frame 20, and a second processing result is obtained, which is the result of applying a second processing to the estimated frame 24. The details of the inspection system 1 will be described below.

[0033] [3. Functions Realized in the Inspection System] Figure 4 is a functional block diagram showing an example of the functions realized in the inspection system 1. As shown in Figure 4, the inspection system 1 realizes the following functions: game processing unit 300, rendering unit 302, rendering information storage unit 304, input frame acquisition unit 306, variation information acquisition unit 308, intermediate frame acquisition unit 310, machine learning model storage unit 312, estimated frame acquisition unit 314, motion information acquisition unit 316, depth information acquisition unit 318, appearance pixel identification unit 320, auxiliary information acquisition unit 322, first processing result acquisition unit 324, second processing result acquisition unit 326, and display unit 328. The game processing unit 300, rendering unit 302, input frame acquisition unit 306, variation information acquisition unit 308, intermediate frame acquisition unit 310, estimated frame acquisition unit 314, motion information acquisition unit 316, depth information acquisition unit 318, appearance pixel identification unit 320, auxiliary information acquisition unit 322, first processing result acquisition unit 324, and second processing result acquisition unit 326 are mainly implemented by the control device 10. The rendering information storage unit 304 and machine learning model storage unit 312 are mainly implemented by the storage device 12. The display unit 328 is mainly implemented by the display device 18.

[0034] [Game Processing Unit] The game processing unit 300 performs various processes related to the game. For example, the game processing unit 300 performs processes such as placing game objects O in the virtual 3D space VS, operating or moving the game objects O, and changing the viewpoint C from which the virtual 3D space VS is viewed, in response to the game development program executed by the control device 10 and the user input received by the operation device 16 (see Figure 5). The game objects O are composed of primitives such as polygons represented by 3D data. The 3D data includes geometric information that indicates the position of vertices, phase information that indicates how the vertices are connected, and attribute information such as color.

[0035] [Rendering Unit] Figure 5 is a diagram illustrating the processing of the rendering unit 302. The rendering unit 302 generates the first to Nth (N is a natural number of 2 or more) input frames 20 by rendering (drawing) three-dimensional data representing one or more game objects O as seen from a predetermined viewpoint C. In other words, the rendering unit 302 functions as an input moving image acquisition means. The rendering unit 302 performs rendering based on the results of various processes performed by the game processing unit 300. Specifically, the rendering unit 302 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes coordinate transformation processing from the view coordinate system to the screen coordinate system (perspective projection), and the perspective projection matrix (camera matrix) used in the coordinate transformation processing has numerical values ​​related to the change in viewpoint C added to it, as will be described later. The rendering unit 302 may also perform rendering based on light source information (luminance value), depth information (depth buffer), texture information, normal information, etc. Here, game software developers may adjust the texture's MIP (Multiplier) according to the estimated number of pixels in the estimated frame 24. This makes it possible to suppress the generation of noise such as moiré patterns in the estimated frame 24.

[0036] Here, the rendering unit 302 generates each input frame 20 by performing rendering such that the viewpoint C changes with each input frame 20. Even if the game processing unit 300 fixes the viewpoint C in a predetermined position, the rendering unit 302 adds a change to the viewpoint C with each input frame 20. As a result, as shown in Figure 5, the position of the displayed game object O changes in each of the input frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 302 applies jitter when generating each input frame 20. Specifically, the rendering unit 302 changes the viewpoint C with each input frame 20 by adding a numerical value corresponding to a size less than one pixel, which differs for each input frame 20, to the perspective projection matrix. The rendering unit 302 changes the viewpoint C with each input frame 20 according to a predetermined rule. For example, the Halton column can be used as such a rule.

[0037] [Rendering Information Storage Unit] The rendering information storage unit 304 stores information necessary for rendering processing in the rendering unit 302 and information obtained as a result of rendering processing. For example, the rendering information storage unit 304 stores the input frame 20. The rendering information storage unit 304 also stores variation information, motion information, and depth information. Details of variation information, motion information, and depth information will be described later. In addition, the rendering information storage unit 304 may store parameters used for coordinate transformation, light source information, texture information, normal information, color temperature information, etc.

[0038] [Machine Learning Model] The machine learning model 200 receives an input video and acquires an output video containing multiple output frames, each having an estimated number of pixels greater than the number of input pixels. Specifically, the machine learning model 200 generates the first to nth intermediate frames 22, each corresponding to the input frame 20 and having an intermediate number of pixels greater than or equal to the number of input pixels, based on each input frame 20. The machine learning model 200 also outputs the nth cumulative feature information 26_n, which represents the characteristics of the first to nth intermediate frames 22, based on the nth intermediate frame 22_n (n=2, 3, ..., N) and the nth-1st auxiliary information 28_n-1, which is based on the nth-1st cumulative feature information 26_n-1 that represents the characteristics of the first to nth intermediate frames 22. Based on each cumulative feature information 26, the machine learning model 200 generates the first to nth estimated frames 24, each having an estimated number of pixels greater than or equal to the number of intermediate pixels and greater than or equal to the number of input pixels. The estimated frames 24 are output frames output from the machine learning model 200. The output video is composed of multiple output frames. In other words, the machine learning model 200 functions as a means for acquiring the output video.

[0039] The machine learning model 200 is trained using multiple training data sets, each containing a training intermediate frame with an intermediate pixel count generated based on the training input frame contained in the training input video, and a training estimated frame contained in the training output video. Various known methods, such as backpropagation, can be used to train the machine learning model 200.

[0040] The machine learning model 200 is, for example, a convolutional neural network (CNN). As the machine learning model 200, known models such as the multilayer structure ResNet with a residual connection mechanism, or the so-called encoder-decoder type U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model 200.

[0041] [Input Frame Acquisition Unit] The input frame acquisition unit 306 acquires the first to the Nth input frames 20. Specifically, the input frame acquisition unit 306 acquires the first to the Nth input frames 20 stored in the rendering information storage unit 304.

[0042] [Variation Information Acquisition Unit] The variation information acquisition unit 308 acquires variation information. The variation information acquisition unit 308 acquires variation information stored in the rendering information storage unit 304. The variation information is information relating to the variation of viewpoint C for each input frame 20 in rendering. Specifically, the variation information is information indicating the amount of variation of viewpoint C between before and after the variation. The information indicating the amount of variation can also be said to be a variation vector indicating the direction and distance of the variation. For example, since the aforementioned Halton column contains information indicating the amount of variation of viewpoint C, that information may be used as variation information.

[0043] [Intermediate Frame Acquisition Unit] The intermediate frame acquisition unit 310 acquires the first to the Nth intermediate frames 22 by generating an intermediate frame 22 that corresponds to the input frame 20 and has an intermediate pixel count equal to or greater than the number of input pixels, based on each input frame 20. In this embodiment, each intermediate frame 22 has an intermediate pixel count greater than the number of input pixels. That is, in this embodiment, each intermediate frame 22 is an enlarged image of the input frame 20 corresponding to the intermediate frame 22.

[0044] Specifically, the intermediate frame acquisition unit 310 obtains the pixel values of the pixels at the positions corresponding to the pixels before the change in the input frame 20 by interpolation based on the change information and each pixel of each input frame 20, and generates each intermediate frame 22. FIG. 6 is a diagram for explaining the processing in the intermediate frame acquisition unit 310. In FIG. 6, the case of obtaining the n-th intermediate frame 22_n is exemplified. For example, as shown in FIG. 6, if the pixel center of a certain pixel in the intermediate frame 22_n to be obtained is P1,0, the intermediate frame acquisition unit 310 determines the coordinates and pixel values of the four pixels closest to P1,0 in the input frame 20_n, namely P'0,0, P'1,0, P'0,1, and P'1,1, and obtains the pixel value of P1,0 by bilinear interpolation. Here, P'1,0 is at a position shifted from P1,0 by the amount of change indicated by the change information. The pixel values of the newly generated pixels by the enlargement process are also obtained in the same manner. In addition to bilinear interpolation, various known methods such as bicubic interpolation and Lanczos interpolation can be used as the interpolation method.

[0045] When rendering is executed such that the viewpoint C varies for each input frame 20, the amount of information in the time series increases. If each input frame 20 obtained in this way (hereinafter referred to as "varying input frame") is used for estimation, a higher-quality estimated frame 24 can be obtained.

[0046] On the other hand, if the varying input frame (or an image obtained by enlarging this) is directly input to the machine learning model 200, there is also a possibility that the estimation accuracy will decrease due to the influence of the variation of the viewpoint C.

[0047] Therefore, in the inspection system 1, as described above, based on the change information and each pixel of each input frame 20, the pixel values of the pixels at the positions corresponding to the pixels before the change in the input frame 20 are obtained by interpolation, and each intermediate frame 22 is generated. As a result, the influence of the variation of the viewpoint C is corrected, and it is possible to suppress the decrease in the estimation accuracy.

[0048] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 314 receives each intermediate frame 22 and acquires the first to Nth estimated frames 24 each having an estimated pixel number greater than the number of input pixels and equal to or greater than the number of intermediate pixels. In the present embodiment, the estimated frame 24 has the same number of estimated pixels as the number of intermediate pixels. More specifically, the estimated frame acquisition unit 314 acquires the nth estimated frame 24_n based on the nth intermediate frame 22_n and the (n - 1)th auxiliary information 28_n - 1.

[0049] [Motion Information Acquisition Unit] The motion information acquisition unit 316 acquires the (n - 1)th motion information, which is information indicating the amount and direction of motion from the (n - 1)th input frame 20_n - 1 to the nth input frame 20_n. Specifically, the (n - 1)th motion information has the same number of pixels as the number of intermediate pixels and is image information (information in bitmap format) indicating the amount and direction of motion of each pixel between the (n - 1)th input frame 20_n - 1 and the nth input frame 20_n. Motion information is also called a motion vector. More specifically, the motion information acquisition unit 316 acquires original motion information having the same number of pixels as the number of input pixels, and executes enlargement and interpolation processing on the original motion information to acquire motion information having the same number of pixels as the number of intermediate pixels.

[0050] [Depth Information Acquisition Unit] The depth information acquisition unit 318 acquires the (n - 1)th depth information indicating the depth of each pixel of the (n - 1)th input frame 20_n - 1 and the nth depth information indicating the depth of each pixel of the nth input frame 20_n. Specifically, the depth information has the same number of pixels as the number of intermediate pixels and is image information (information in bitmap format). Depth information is also called a depth buffer, Z-buffer, or depth buffer. More specifically, the depth information acquisition unit 318 acquires original depth information having the same number of pixels as the number of input pixels, and executes enlargement and interpolation processing on the original depth information to acquire depth information having the same number of pixels as the number of intermediate pixels.

[0051] [Appearing Pixel Identification Unit] The appearing pixel identification unit 320 identifies the nth appearing pixel 222_n, which is a pixel in the nth intermediate frame 22_n that displays all or part of the game object O that was not displayed in the nth intermediate frame 22_n-1, based on the n-th depth information and the nth depth information (see Figure 3). Specifically, the appearing pixel identification unit 320 identifies the nth appearing pixel 222_n based on the difference between the n-th depth information and the nth depth information. Alternatively, the appearing pixel identification unit 320 may identify the nth appearing pixel 222_n based on the n-th perspective projection matrix relating to the n-th intermediate frame 22_n-1 and the nth perspective projection matrix relating to the nth intermediate frame 22_n. Furthermore, the appearing pixel identification unit 320 may identify the nth appearing pixel 222_n using the n-th motion information. More specifically, the occurrence pixel identification unit 320 identifies the nth occurrence pixel 222_n and generates the nth occurrence pixel information, which is image information indicating the position of the nth occurrence pixel 222_n.

[0052] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 322 acquires the n-th auxiliary information 28_n-1 by applying motion compensation to the n-th cumulative feature information 26_n-1 based on the n-th motion information. Motion compensation refers to the process of moving the pixel at position x in the n-th cumulative feature information 26_n to position x' when, for example, a pixel at position x in the n-th intermediate frame 22_n-1 has moved to position x' in the nth intermediate frame 22_n (see Figure 3). That is, the auxiliary information acquisition unit 322 acquires the n-th auxiliary information 28_n-1 by setting the pixel value of one or more pixels in the n-th cumulative feature information 26_n-1 to the pixel at the position where the pixel has moved according to the amount and direction of the pixel's movement, based on the n-th motion information.

[0053] If there is movement of the game object O between the nth input frame 20_n and the (n-1)th input frame 20_n-1, and the nth estimated frame 24_n is obtained using the nth intermediate frame 22_n and the (n-1)th cumulative feature information 26_n-1 as they are, a ghosting phenomenon may occur in the output nth estimated frame 24_n, where the afterimage of the game object O that was displayed in the nth intermediate frame 22_n is displayed.

[0054] Therefore, in the inspection system 1, as described above, motion compensation is applied to the n-th-first cumulative feature information 26_n-1 based on the n-th-first motion information to obtain the n-th-first auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this n-th-first auxiliary information 28_n-1 is input to the cumulative feature information output layer 202. This makes it possible to suppress the ghosting phenomenon described above.

[0055] Furthermore, the auxiliary information acquisition unit 322 acquires the n-th auxiliary information 28_n-1 by replacing the pixel value of the nth occurrence pixel 222_n in the n-th cumulative feature information 26_n-1 with a predetermined value. Specifically, the auxiliary information acquisition unit 322 acquires the n-th auxiliary information 28_n-1 by replacing the pixel value of the nth occurrence pixel 222_n in the n-th cumulative feature information 26_n-1 with a predetermined value based on the nth occurrence pixel information. The predetermined value may be a constant value such as 0 (black), or it may be the pixel value of the nth occurrence pixel 222_n in the nth intermediate frame 22_n.

[0056] If, in the nth input frame 20_n, all or part of the game object O that was not displayed in the (n-th)th input frame 20_n-1 is displayed, and the nth estimated frame 24_n is obtained using the nth intermediate frame 22_n and the (n-th)th cumulative feature information 26_n-1 as they are, the aforementioned ghosting phenomenon may occur in the output nth estimated frame 24_n.

[0057] Therefore, the inspection system 1 identifies the nth occurrence pixel 222_n, which is a pixel in the nth intermediate frame 22_n that displays all or part of the game object O that was not displayed in the (n-th)th intermediate frame 22_n-1, and obtains the (n-th)th auxiliary information 28_n-1 by replacing the pixel value of the nth occurrence pixel 222_n in the (n-th)th cumulative feature information 26_n-1 with a predetermined value. This makes it possible to suppress the ghosting phenomenon described above.

[0058] [Machine Learning Model Storage Unit] The machine learning model storage unit 312 stores the machine learning model 200. Specifically, the machine learning model storage unit 312 stores the parameters of the machine learning model 200 (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weights of each node).

[0059] [First Processing Result Acquisition Unit] The first processing result acquisition unit 324 applies a predetermined image processing, the first processing, to a first processing frame, which is one of the multiple input frames 20, and acquires the first processing result. For example, the first processing is the process of generating an image to which a bloom effect (an effect that depicts the appearance of light leaking out and spreading from bright areas of an image) has been applied. The first processing is performed by referring to various parameters acquired by the rendering unit 302. Specifically, first, the first processing result acquisition unit 324 sets a bloom mask for each pixel of the input frame 20_n included in the input video by referring to reference information (hereinafter referred to as the first reference information) which includes light source information (information representing the brightness value set for each pixel) and depth information (also called a depth buffer). Here, the depth information included in the first reference information is image information having the same number of pixels as the number of input pixels, and is the original depth information described above. Similarly, the light source information included in the first reference information is image information having the same number of pixels as the number of input pixels. The bloom mask is set so that the bloom effect is greater for pixels with higher brightness values. Furthermore, the Bloommark is set so that the greater the depth information (the further away from viewpoint C), the smaller the bloom effect. The first processing result acquisition unit 324 then calculates the color information to be added to each pixel based on the bloom mask, and obtains image information with the bloom effect applied by adding the calculated color information to each pixel of the input frame 20_n. This makes it possible to represent the situation where bright light exceeds the exposure limit of the camera taking the picture of the scene. Note that when setting the bloom mask, the first processing result acquisition unit 324 may refer to fluctuation information and motion information in addition to brightness values ​​and depth information.

[0060] Furthermore, the first processing result acquisition unit 324 acquires a first processing result having a number of pixels greater than the input number of pixels (estimated number of pixels) based on the image information to which the bloom effect has been applied. Specifically, the first processing result is generated by performing scaling and interpolation processing on the image information to which the bloom effect has been applied. The first processing result may be stored in the rendering information storage unit 304.

[0061] The upper part of Figure 7 shows an example of the first processing result displayed on the display unit 328. In the embodiment shown in Figure 7, a bloom effect additive processing is performed on the pixels surrounding the light-emitting portion of the streetlights. Here, since brightness values ​​and depth information are referenced, for the light-emitting portion of each streetlight, color information is added to a wider area of ​​pixels the brighter the streetlight and the closer it is to viewpoint C (i.e., closer to the viewer).

[0062] [Second Processing Result Acquisition Unit] The second processing result acquisition unit 326 applies the second processing corresponding to the first processing to the second processing frame, which is the output frame corresponding to the first processing frame, and acquires the second processing result. Specifically, the second processing is an image processing that produces the same visual effect as the first processing, differing only in the number of pixels of the various parameters referenced. For example, if the first processing is a process that generates an image with a bloom effect, the second processing is also a process that generates an image with a bloom effect. The input frame 20 input to the first processing result acquisition unit 324 has an input number of pixels, while the estimated frame 24 input to the second processing result acquisition unit 326 has an estimated number of pixels. Therefore, the second processing result acquisition unit 326 acquires reference information (hereinafter referred to as second reference information) having the same number of pixels as the estimated number of pixels by performing scaling and interpolation processing on the first reference information referenced in the first processing.

[0063] The second processing frame is an output frame generated by the estimated frame acquisition unit 314 when the first processing frame is input to the intermediate frame acquisition unit 310. For example, if the first processing frame is input frame 20_n, the second processing frame is estimated frame 24_n. The second processing result acquisition unit 326 generates an image with a bloom effect applied to the estimated frame 24_n using the second reference information, which has undergone scaling and interpolation processing. The method for calculating the color information added by the bloom effect is the same as that used by the first processing result acquisition unit 324.

[0064] The lower part of Figure 7 shows an example of the second processing result displayed on the display unit 328. In the embodiment shown in Figure 7, a bloom effect additive processing is performed on the pixels surrounding the light-emitting portion of the streetlights. Here, since brightness values ​​and depth information are referenced, for each light-emitting portion of a streetlight, the brighter the streetlight and the closer it is to viewpoint C (i.e., closer), the wider the area of ​​pixels to which color information is added. However, as shown in Figure 7, the way the bloom effect manifests differs between the first processing result and the second processing result. The area to which color information is added in the second processing result is wider than in the first processing result. In other words, the second processing result shows a stronger bloom effect than the first processing result.

[0065] The first reference information is set according to the game developer's intentions for each input frame 20. However, the estimated frame 24 generated based on the input frame 20 is not simply image information that has been enlarged and interpolated. On the other hand, the second reference information is image information that has been simply enlarged and interpolated relative to the first reference information. Therefore, the relationship between the estimated frame 24 and the second reference information may differ from the game developer's intentions. In the example in Figure 7, the result of the second processing shows a stronger bloom effect than the developer intended. In particular, when the results of the first and second processing are moving images, problems may occur such as streetlights that should be lit flickering or colors changing.

[0066] [Display Unit] The display unit 328 compares and displays the first processing result and the second processing result. Specifically, as shown in Figure 7, for example, the display unit 328 displays the first processing result and the second processing result side by side. The first processing result is the result of the image processing that was originally intended by the game creator. Therefore, by displaying the first processing result and the second processing result side by side, it is easy to confirm whether the second processing result reflects the image processing result intended by the game creator.

[0067] [Modification 1] The above describes a case where the first and second processes generate an image with a bloom effect applied, but it is not limited to this. For example, the first and second processes may be processes that generate a histogram. Figure 8 is an example of comparing the results of the first and second processes, which are histograms.

[0068] The first processing result acquisition unit 324 generates a histogram, which is the first processing result, by counting the frequency of each brightness (pixel value) for each color in the first processing frame, which is one of the multiple input frames 20 (Figure 8, top). The second processing result acquisition unit 326 generates a histogram, which is the second processing result, by counting the frequency of each brightness (pixel value) for each color in the second processing frame, which is the output frame corresponding to the first processing frame (Figure 8, bottom). If the number of pixels in the input frame 20 and the estimated frame 24 are different, the histograms of the first and second processing results are normalized. The estimated frame 24 is generated based on the input frame 20, but as shown in Figure 8, the distribution of brightness (pixel values) differs between the histogram of the first and second processing results. If the estimated frame 24 and the input frame 20 are displayed directly on the display unit 328, the difference may be difficult to discern with the naked eye. However, as shown in Figure 8, by displaying the data using a histogram, it is possible to check if there are differences in the distribution of brightness (pixel values). Such differences in histograms can cause bloom effects in ways not intended by the creator, as described above.

[0069] Furthermore, if the first and second processing results can be displayed overlaid, such as in a histogram, the display unit 328 may display the first and second processing results overlaid, as shown in Figure 9. By displaying them overlaid, the differences between the first and second processing results can be more easily confirmed. In addition, the display unit 328 may display the difference between the first and second processing results.

[0070] [Modification 2] The first and second processes may be processes that generate a heat map. The process of generating a heat map is a process that visualizes the strength of numerical data using color in order to visualize the data.

[0071] The first processing result acquisition unit 324 acquires color temperature information for each pixel of a first processing frame, which is one of the multiple input frames 20, and generates a first processing result, which is image information in which a predetermined color is assigned to each color temperature (Figure 10, upper part). The second processing result acquisition unit 326 acquires color temperature information for each pixel of a second processing frame, which is an output frame corresponding to the first processing frame, and generates a second processing result, which is image information in which a predetermined color is assigned to each color temperature (Figure 10, lower part).

[0072] In some cases, the brightness information (pixel values) in the estimated frame 24 may be amplified compared to the input frame 20. In such cases, even if the high-brightness areas of the input frame 20 and the corresponding areas of the estimated frame 24 are displayed directly on the display device 18, the difference in brightness may be difficult to discern with the naked eye. In such cases, by comparing and displaying the first processing result, which is a heat map, and the second processing result, which is also a heat map, it is easy to confirm whether the brightness has changed.

[0073] [Modification 3] The first and second processes may be processes that generate an image with a depth-of-field effect applied. The depth-of-field effect is an effect that blurs the image according to the distance from the focal point. The first processing result acquisition unit 324 generates a first processing result, which is image information with a depth-of-field effect applied, for a first processing frame, which is one of the multiple input frames 20, using first reference information that includes depth information. The second processing result acquisition unit 326 generates a second processing result, which is image information with a depth-of-field effect applied, for a second processing frame, which is an output frame corresponding to the first processing frame, using second reference information that includes depth information. The display unit 328 displays the first processing result and the second processing result side by side, making it easy to confirm whether the second processing result reflects the image processing result intended by the game creator.

[0074] [Modification 4] The first and second processes may be processes that generate an image with a motion blur effect applied. The motion blur effect is an effect that adds afterimages when rendering moving objects. The first processing result acquisition unit 324 generates a first processing result, which is image information with a depth of field effect applied, for a first processing frame, which is one of the multiple input frames 20, using first reference information that includes depth information and motion information (here, original motion information having the same number of pixels as the input frame 20). The second processing result acquisition unit 326 generates a second processing result, which is image information with a depth of field effect applied, for a second processing frame, which is an output frame corresponding to the first processing frame, using second reference information that includes depth information and motion information (here, motion information having the same number of pixels as the estimated frame 24). The display unit 328 displays the first processing result and the second processing result side by side, making it easy to confirm whether the second processing result reflects the image processing result intended by the game creator.

[0075] The first and second processes described above are merely examples, and other known image processing methods may be applied to this disclosure. For example, the first and second processes may be processes that generate a second processing result, which is image information in which the exposure value has been changed for the first processing frame.

[0076] [4. Processing performed by the inspection system] Figures 11 and 12 are flowcharts showing an example of the processing flow (game program inspection method) performed by the inspection system 1. The processing shown in Figures 11 and 12 is performed by the control device 10 operating according to the program stored in the storage device 12. (1) Processing when n=1

[0077] First, the control device 10 acquires the first input frame 20_1 (S1100). Based on the first input frame 20_1, the control device 10 acquires the first intermediate frame 22_1 (S1102). Then, based on the first intermediate frame 22_1 and given auxiliary information, the control device 10 acquires the first estimated frame 24_1 and the first cumulative feature information 26_1 (S1104). (2) Processing when n≧2

[0078] The control device 10 acquires the nth input frame 20_n (S1106). Based on the nth input frame 20_n, the control device 10 acquires the nth intermediate frame 22_n (S1108).

[0079] Next, the control device 10 acquires the (n-1)th motion information (S1110). The control device 10 also acquires the (n-1)th depth information and the nth depth information (S1112), and identifies the nth occurrence pixel 222_n based on the (n-1)th depth information and the nth depth information (S1114). The control device 10 acquires the (n-1)th auxiliary information 28_n-1 based on the (n-1)th cumulative feature information 26_n-1, the (n-1)th motion information, and the nth occurrence pixel 222_n (S1116). Then, the control device 10 inputs the nth intermediate frame 22_n and the (n-1)th auxiliary information 28_n-1 into the machine learning model 200 and acquires the nth estimated frame 24_n and the nth cumulative feature information 26_n (S1118). The control device 10 determines whether or not the next frame exists (S1120). If it determines that the next frame exists (S1120; Y), it increments n = n + 1 and repeats the processes of S1106 to S1118. If the control device 10 determines that the next frame does not exist (S1120; N), it proceeds to S1200.

[0080] Next, the first processing result acquisition unit 324 applies a predetermined image processing, which is a first processing, to a first processing frame (here, for example, input frame 20_n) which is one of the plurality of input frames 20, and acquires a first processing result (S1200). The second processing result acquisition unit 326 applies a second processing, which corresponds to the first processing, to a second processing frame (here, for example, estimated frame 24_n) which is an output frame corresponding to the first processing frame, and acquires a second processing result (S1202). Then, the display unit 328 compares the first processing result and the second processing result and displays them (S1204).

[0081] [5. Summary] According to the inspection system 1 of this embodiment described above, the first processing result and the second processing result are displayed in comparison. Therefore, it is possible to easily determine whether any visual effects unintended by the game creator appear in the estimated frame 24. If, as a result of the determination, unintended visual effects appear, they can be improved by retraining the machine learning model 200.

[0082] Furthermore, the present invention is not limited to the embodiments described above. Also, the specific strings and numbers shown above and in the drawings are illustrative examples and the invention is not limited to these.

[0083] For example, in this embodiment, we have illustrated the case where the number of intermediate pixels is greater than the number of input pixels, and the number of intermediate pixels and the number of estimated pixels are the same. However, the number of intermediate pixels and the number of input pixels may be the same, and the number of estimated pixels may be greater than the number of intermediate pixels. In other words, the intermediate frame 22 does not necessarily have to be an enlarged version of the input frame 20. [6. [Note] (1) A game program inspection system comprising: a processor; a storage unit that stores instructions executed by the processor and a machine learning model; and a display unit, wherein at least one processor acquires an input video including a plurality of input frames having a predetermined number of input pixels; applies a first processing which is a predetermined image processing to a first processing frame which is one of the plurality of input frames and obtains a first processing result; inputs the input video to the machine learning model and obtains an output video including a plurality of output frames which have an estimated number of pixels greater than the number of input pixels; applies a second processing which corresponds to the first processing to a second processing frame which is the output frame corresponding to the first processing frame and obtains a second processing result; the display unit compares and displays the first processing result and the second processing result; and the machine learning model is trained using training data which includes a training input video and a training output video. (2) The game program inspection system according to (1), wherein the first processing and the second processing are processes that generate an image with a bloom effect applied. (3) The game program inspection system according to (1), wherein the first and second processes are processes for generating a heatmap. (4) The game program inspection system according to (1), wherein the first and second processes are processes for generating a histogram. (5) The game program inspection system according to (1), wherein the first and second processes are processes for generating an image with a depth-of-field effect applied. (6) The game program inspection system according to (1), wherein the first and second processes are processes for generating an image with a motion blur effect applied.(7) The game program inspection system according to any one of (1) to (6), wherein the display unit displays the first processing result and the second processing result side by side. (8) The game program inspection system according to (4), wherein the display unit displays the first processing result and the second processing result superimposed on each other. (9) A game program inspection system, which is trained using a plurality of training data, the machine learning model generates first to nth intermediate frames based on each input frame, each corresponding to the input frame and having an intermediate pixel count equal to or greater than the number of input pixels; generates first to nth estimated frames based on each intermediate frame, each having an estimated pixel count greater than or equal to the number of input pixels and greater than or equal to the number of intermediate pixels; outputs the nth cumulative feature information indicating the characteristics of the first to nth intermediate frames based on the nth intermediate frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information indicating the characteristics of the first to nth intermediate frames; outputs the nth estimated frame based on the nth cumulative feature information; and each learning intermediate frame having the intermediate pixel count generated based on the learning input frame included in the learning input video, and the learning estimated frame included in the learning output video, respectively. (10) A method for testing a game program, wherein the processor acquires an input video including a plurality of input frames having a predetermined number of input pixels, applies a first processing which is a predetermined image processing to a first processing frame which is one of the plurality of input frames and obtains a first processing result, inputs the input video to the machine learning model and obtains an output video including a plurality of output frames which have an estimated number of pixels greater than the number of input pixels, applies a second processing which corresponds to the first processing to a second processing frame which is the output frame corresponding to the first processing frame and obtains a second processing result, the display unit compares and displays the first processing result and the second processing result, and the machine learning model is trained using training data which includes a training input video and a training output video.(11) A program for causing a computer to function as follows: input video acquisition means for acquiring an input video including a plurality of input frames having a predetermined number of input pixels; first processing result acquisition means for applying a first processing, which is a predetermined image processing, to a first processing frame, which is one of the plurality of input frames, and acquiring a first processing result; output video acquisition means for inputting the input video to the machine learning model and acquiring an output video including a plurality of output frames having an estimated number of pixels greater than the number of input pixels; second processing result acquisition means for applying a second processing, which is the output frame corresponding to the first processing frame, to a second processing frame, which is the output frame corresponding to the first processing frame, and acquiring a second processing result; and display means for comparing and displaying the first processing result and the second processing result, wherein the machine learning model is trained using training data including training input video and training output video.

Claims

1. A game program testing system comprising: a processor; a storage unit that stores instructions executed by the processor and a machine learning model; and a display unit, wherein at least one processor acquires an input video including a plurality of input frames having a predetermined number of input pixels; applies a first processing, which is a predetermined image processing, to a first processing frame that is one of the plurality of input frames and obtains a first processing result; inputs the input video to the machine learning model and acquires an output video including a plurality of output frames having an estimated number of pixels greater than the number of input pixels; applies a second processing, which is the output frame corresponding to the first processing frame, to a second processing result corresponding to the first processing, and obtains a second processing result; the display unit compares and displays the first processing result and the second processing result; and the machine learning model is trained using training data including training input video and training output video.

2. The game program inspection system according to claim 1, wherein the first and second processes are processes that generate an image with a bloom effect applied.

3. The game program inspection system according to claim 1, wherein the first and second processes are processes for generating a heat map.

4. The game program inspection system according to claim 1, wherein the first and second processes are processes for generating a histogram.

5. The game program inspection system according to claim 1, wherein the first and second processes are processes that generate an image to which a depth-of-field effect has been applied.

6. The game program inspection system according to claim 1, wherein the first and second processes are processes that generate an image to which a motion blur effect has been applied.

7. The game program inspection system according to any one of claims 1 to 6, wherein the display unit displays the first processing result and the second processing result side by side.

8. The game program inspection system according to claim 4, wherein the display unit displays the first processing result and the second processing result superimposed on each other.

9. A game program inspection system, the machine learning model is trained using a plurality of training data, each including: a first to nth intermediate frame having an intermediate pixel count equal to or greater than the number of input pixels, based on each of the input frames; a first to nth estimated frame having an estimated pixel count greater than or equal to the number of input pixels, based on each of the intermediate frames; an nth cumulative feature information indicating the characteristics of the first to nth intermediate frames, based on the nth intermediate frame (n = 2, 3, ..., N) and the nth-1st auxiliary information based on the nth-1st cumulative feature information indicating the characteristics of the first to nth intermediate frames; an nth estimated frame based on the nth cumulative feature information; and a learning intermediate frame having the intermediate pixel count generated based on the learning input frame included in the learning input video, and a learning estimated frame included in the learning output video, respectively.

10. A method for testing a game program, wherein a processor acquires an input video including a plurality of input frames having a predetermined number of input pixels, applies a predetermined image processing to a first processing frame which is one of the plurality of input frames, and obtains a first processing result, inputs the input video to a machine learning model, and obtains an output video including a plurality of output frames which have an estimated number of pixels greater than the number of input pixels, applies a second processing corresponding to the first processing to a second processing frame which is the output frame corresponding to the first processing frame, and obtains a second processing result, the display unit compares and displays the first processing result and the second processing result, and the machine learning model is trained using training data including a training input video and a training output video.

11. A program for causing a computer to function as follows: input video acquisition means for acquiring an input video including a plurality of input frames having a predetermined number of input pixels; first processing result acquisition means for applying a predetermined image processing to a first processing frame which is one of the plurality of input frames and acquiring a first processing result; output video acquisition means for inputting the input video to a machine learning model and acquiring an output video including a plurality of output frames having an estimated number of pixels greater than the number of input pixels; second processing result acquisition means for applying a second processing corresponding to the first processing to a second processing frame which is the output frame corresponding to the first processing frame and acquiring a second processing result; and display means for comparing and displaying the first processing result and the second processing result, wherein the machine learning model is trained using training data including training input video and training output video.

Images