Inspection system, inspection method, and program
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-04
- Publication Date
- 2026-08-13
Smart Images
Figure JP2025003611_13082026_PF_FP_ABST
Abstract
Description
Testing systems, testing methods, programs
[0001] This invention relates to an inspection system, an inspection method, and a program.
[0002] Conventionally, a technique (super-resolution) has been known that uses machine learning models to estimate high-resolution still images based on low-resolution still images (see Non-Patent Document 1 below).
[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014
[0004] The inventors of this application are considering using a machine learning model with a recursive structure to obtain high-quality estimated frames. In the estimation of high-quality frames by such a machine learning model with a recursive structure, not only information from the current frame to be processed but also information from past frames of that frame contributes. By contributing information from past frames in addition to the current frame to the estimation, it is expected that the estimation accuracy of the machine learning model will improve. In such a technology, based on viewpoint information and motion vectors related to the camera viewpoint, areas that were hidden in past frames but appeared in the current frame are identified as disocclusion regions, and the accuracy of the image quality can be improved by using this as auxiliary information. However, if there is an error in the viewpoint information, it may affect the estimation result. In this case, it is best to reset the viewpoint information to correct information, but it is not easy to check whether the viewpoint information is accurate or not.
[0005] This disclosure aims to provide an inspection system, inspection method, and inspection program that can inspect the accuracy of viewpoint information in an image processing method that estimates high-quality moving images using information from past frames.
[0006] The inspection system relating to this disclosure is an image processing system that obtains first to nth (n is a natural number of 2 or more) input frames by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint in a virtual space, and inputs the nth (n=2, 3, ..., N) input frame, the nth-1st feature information showing the characteristics of the nth-1st input frame, and the nth-1st auxiliary information based on viewpoint information relating to the viewpoint into a machine learning model, thereby obtaining the nth estimated frame output from the machine learning model, and the inspection system includes at least one processor that inspects the accuracy of the viewpoint information, wherein the at least one processor obtains the nth-1st viewpoint information corresponding to the nth-1st input frame, obtains the nth viewpoint information corresponding to the nth input frame, and obtains relative motion information showing the relative movement of the viewpoint and the object on the screen plane accompanying the movement from the nth-1st input frame to the nth input frame, and An inspection system that, based on the nth viewpoint information, obtains the nth viewpoint coordinate system position information of the object in the nth viewpoint coordinate system with the viewpoint in the nth viewpoint information as the origin; converts the nth viewpoint coordinate system position information into the nth position information in the virtual space; converts the nth position information into the nth-1 viewpoint coordinate system position information in the nth-1 viewpoint coordinate system with the viewpoint in the nth-1 viewpoint information as the origin, based on the n-1 viewpoint information; obtains a reprojection input frame by projecting the object onto the screen plane based on the nth viewpoint coordinate system position information; obtains viewpoint motion information showing the movement of the object on the screen plane due to the movement of the viewpoint based on the nth input frame and the reprojection input frame; obtains object motion information showing the movement of the object in the virtual space based on the relative motion information and the viewpoint motion information; and displays an object motion image based on the object motion information on the display unit.
[0007] This figure shows an example of the hardware configuration of the image processing system and the inspection system. This figure schematically shows the processing of the image processing system. This figure schematically shows the processing of the image processing system. This is a functional block diagram showing an example of the functions realized by the image processing system. This figure explains the processing of the rendering unit. This figure explains the processing in the input frame acquisition unit. This figure shows an overview of the back-projection frame generation process in the inspection system. This figure shows the procedure for checking the accuracy of viewpoint information in the inspection system. This figure shows the procedure for checking the accuracy of viewpoint information in the inspection system. This is a functional block diagram showing an example of the functions realized by the inspection system. This is a flowchart showing an example of the processing flow executed in the inspection system.
[0008] Hereinafter, an example of an embodiment of the image processing system 1 and inspection system 100 according to the present invention will be described with reference to the drawings.
[0009] [1. Hardware Configuration of the Image Processing System] Figure 1 shows an example of the hardware configuration of the image processing system 1. The image processing system 1 is a computer such as a game console (game machine).
[0010] As shown in Figure 1, the image processing system 1 includes a control unit 10 containing at least one processor, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19.
[0011] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1, for example. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images to a frame buffer based on graphics commands and data supplied from the CPU.
[0012] The memory unit 12 includes, for example, a main memory such as ROM or RAM and an auxiliary memory such as an HDD or SSD. The memory unit 12 stores instructions executed by the processor and the machine learning model 200. Specifically, the memory unit 12 stores programs executed by the control unit 10 and the trained machine learning model 200. The machine learning model 200 is trained using training data that includes training input videos and training output videos. The memory unit 12 has an area reserved for a frame buffer on which images are rendered by the GPU. In addition to programs for realizing various functions of the image processing system 1 described later, the memory unit 12 may also store, for example, a game program (game software).
[0013] The communication unit 14 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.
[0014] The operation unit 16 is a user interface such as a keyboard, mouse, or game console controller, and receives user input and outputs a signal indicating the content of that input to the control unit 10.
[0015] The display unit 18 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to the instructions of the control unit 10.
[0016] The audio output unit 19 is, for example, a speaker, and outputs the audio indicated by the audio data generated by the image processing system 1.
[0017] In addition to the devices described above, the image processing system 1 may also include an optical disc drive for reading optical discs such as DVD-ROMs and Blu-ray® discs, and a USB (Universal Serial Bus) port.
[0018] [2. Overview of the Image Processing System] Figures 2 and 3 are schematic diagrams illustrating the processing of the image processing system 1. In this embodiment, an example is given in which the image processing system 1 is used to improve the image quality of gameplay videos in a game. Gameplay videos are videos generated in response to the game program executed by the control unit 10 and user input received by the operation unit 16, and consist of multiple still images (frames) which are time-series data. The processing performed by the image processing system 1 is mainly as follows.
[0019] (1) Generation of Processing Frames First, the image processing system 1 generates an image (processing frame) in which a game object is drawn by rendering three-dimensional data showing one or more game objects viewed from a predetermined viewpoint. This processing frame is an image having a predetermined number of pixels (initial number of pixels) and a predetermined image quality (initial image quality). Processing frames are generated at predetermined time intervals. The number of pixels in a processing frame is, for example, 1920 × 1080 (1080p). Each generated processing frame is not displayed directly on the display unit 18, but is temporarily stored in the storage unit 12 and used for subsequent processing. In the following explanation, the processing of the nth (2 ≤ n ≤ N, n and N are natural numbers greater than or equal to 2) processing frame 20_n will be mainly used as an example, but similar processing is performed on other processing frames as well (i.e., n = 2, 3, ..., N).
[0020] (2) Acquisition of input frame The image processing system 1 acquires a frame (input frame) 22_n having a number of pixels greater than the initial number of pixels (input number of pixels) based on the acquired processing target frame 20_n. The input number of pixels is, for example, 3840 x 2160 (4K). Specifically, the input frame 22_n is generated by performing scaling and interpolation processing on the processing target frame 20_n.
[0021] Here, it should be noted that although the input frame 22_n has more pixels than the processing frame 20_n, its image quality is not necessarily sufficiently improved. In other words, the image quality of a frame is not simply determined by the number of pixels (high resolution). The image quality of a frame may be evaluated based on, for example, the high signal-to-noise ratio, the high fidelity of spatial frequencies, and the high temporal stability (the low level of artifacts and flickering when multiple frames are displayed consecutively), either individually or as a combination of these factors, when compared to a reference frame.
[0022] (3) Acquisition of Estimated Frame The image processing system 1 inputs the input frame 22_n to the machine learning model 200 and acquires the estimated frame 24_n. The estimated frame 24_n is an image with the same number of pixels as the input number of pixels (estimated number of pixels) and an image quality that is equal to or better than the initial image quality (estimated image quality). Here, in addition to the input frame 22_n, the machine learning model 200 receives the (n-1)th auxiliary information 28_n-1 (see Figures 2 and 3). The auxiliary information 28_n-1 is information based on the (n-1)th cumulative feature information 26_n-1 which shows the features of the first to (n-1)th input frames 22.
[0023] (4) The machine learning model 200 for acquiring cumulative feature information has a cumulative feature information output layer 202 that receives input frames 22_n and auxiliary information 28_n-1 as inputs and outputs the nth cumulative feature information 26_n which represents the features of the first to nth input frames 22 (see Figure 2). The image processing system 1 acquires the nth cumulative feature information 26_n.
[0024] The acquired nth cumulative feature information 26_n is input to the estimation frame output layer 204, and the estimation frame output layer 204 outputs the nth estimated frame 24_n. The acquired nth cumulative feature information 26_n is also stored in the memory unit 12 and used to estimate the next processing target frame (the (n+1)th processing target frame) 20_n+1, which corresponds to the estimated frame 24_n+1.
[0025] (5) Acquisition of auxiliary information The n-1st cumulative feature information 26_n-1 is, as described above, information that shows the characteristics of the 1st to n-1st input frames 22 (and by extension, the 1st to n-1st processing target frames 20). If the cumulative feature information 26_n-1, which has accumulated information on past processing target frames 20 in this way, is used to estimate the nth estimated frame 24_n, the amount of information available for estimation increases, so a high-quality estimated frame 24_n can be obtained.
[0026] If there is movement or other action in the displayed game object between the (n-1)th processing frame 20_n-1 and the nth processing frame 20_n, and the nth input frame 22_n and the cumulative feature information 26_n-1 are directly input to the machine learning model 200, a phenomenon may occur where a ghost image of the game object displayed in the (n-1)th processing frame 20_n-1 is displayed.
[0027] Therefore, the image processing system 1 obtains the n-th auxiliary information 28_n-1 by applying various corrections, described later, to the cumulative feature information 26_n-1, based on information obtained during rendering (such as motion vectors and depth buffers) (see Figures 2 and 3). The obtained n-th auxiliary information 28_n-1 is input to the machine learning model 200 along with the nth input frame 22_n, as described above, and is used to estimate the nth estimated frame 24_n.
[0028] [3. Functions Realized by the Image Processing System] Figure 4 is a functional block diagram showing an example of the functions realized by the image processing system 1. As shown in Figure 4, the image processing system 1 realizes a game processing unit 400, a rendering unit 402, a rendering information storage unit 404, a processing target frame acquisition unit 406, a variation information acquisition unit 408, an input frame acquisition unit 410, a machine learning model storage unit 412, an estimated frame acquisition unit 414, an auxiliary information acquisition unit 416, a motion vector acquisition unit 418, a depth information acquisition unit 420, and an occurrence pixel identification unit 422.
[0029] The game processing unit 400, rendering unit 402, processing target frame acquisition unit 406, variation information acquisition unit 408, input frame acquisition unit 410, estimated frame acquisition unit 414, auxiliary information acquisition unit 416, motion vector acquisition unit 418, depth information acquisition unit 420, and occurrence pixel identification unit 422 are mainly implemented by the control unit 10. The rendering information storage unit 404 and machine learning model storage unit 412 are mainly implemented by the storage unit 12. Note that the game processing unit 400, rendering unit 402, and rendering information storage unit 404 are functions provided by the game software.
[0030] [Game Processing Unit] The game processing unit 400 performs various processes related to the game. For example, the game processing unit 400 performs processes such as placing game objects O in the virtual 3D space VS, operating or moving the game objects O, and changing the viewpoint C from which the virtual 3D space VS is viewed, in response to the game program executed by the control unit 10 or the user input received by the operation unit 16. The game objects O are composed of primitives such as polygons represented by 3D data. The 3D data includes geometric information that indicates the position of vertices, phase information that indicates how the vertices are connected, and attribute information such as color.
[0031] [Rendering Unit] Figure 5 is a diagram illustrating the processing of the rendering unit 402. The rendering unit 402 generates the first to Nth (N is a natural number of 2 or more) frames 20 to be processed by rendering (drawing) three-dimensional data representing one or more game objects O as seen from a predetermined viewpoint C. The rendering unit 402 performs rendering based on the results of various processes performed by the game processing unit 400. Specifically, the rendering unit 402 performs vertex processing (vertex shading) and pixel processing (pixel shading) based on three-dimensional data representing game objects O placed in the virtual three-dimensional space VS. The vertex processing includes coordinate transformation processing from the view coordinate system to the screen coordinate system (perspective projection), and the perspective projection matrix (camera matrix) used in the coordinate transformation processing has a numerical value related to the change in viewpoint C added to it, as will be described later. The rendering unit 402 may also perform rendering based on light source information, depth information (depth buffer), texture information, normal information, etc.
[0032] Here, the rendering unit 402 generates each processing target frame 20 by performing rendering such that the viewpoint C changes with each processing target frame 20. Even if the game processing unit 400 fixes the viewpoint C in a predetermined position, the rendering unit 402 adds a change to the viewpoint C with each processing target frame 20. As a result, as shown in Figure 5, the position of the displayed game object O changes in each of the processing target frames 20_n, 20_n+1, and 20_n+2. In other words, the rendering unit 402 applies jitter when generating each processing target frame 20. Specifically, the rendering unit 402 changes the viewpoint C with each processing target frame 20 by adding a numerical value corresponding to a size less than one pixel, which differs for each processing target frame 20, to the perspective projection matrix. The rendering unit 402 changes the viewpoint C with each processing target frame 20 according to a predetermined rule. For example, the Halton column can be used as such a rule.
[0033] [Rendering Information Storage Unit] The rendering information storage unit 404 stores information necessary for rendering processing in the rendering unit 402 and information obtained as a result of rendering processing. For example, the rendering information storage unit 404 stores the frame 20 to be processed. The rendering information storage unit 404 also stores fluctuation information, motion vectors, and depth information. In addition, the rendering information storage unit 404 may store parameters used for coordinate transformation, light source information, texture information, normal information, etc.
[0034] [Processing Frame Acquisition Unit] The processing frame acquisition unit 406 acquires the first to the Nth processing frame 20. Specifically, the processing frame acquisition unit 406 acquires the first to the Nth processing frame 20 stored in the rendering information storage unit 404.
[0035] [Variation Information Acquisition Unit] The variation information acquisition unit 408 acquires variation information. The variation information acquisition unit 408 acquires variation information stored in the rendering information storage unit 404. Specifically, the variation information is information that shows the amount of change of viewpoint C between before and after the change. The information that shows the amount of change can also be said to be a variation vector that shows the direction and distance of the change. For example, since the aforementioned Halton column contains information that shows the amount of change of viewpoint C, that information may be used as variation information.
[0036] [Input Frame Acquisition Unit] The input frame acquisition unit 410 acquires the first to the Nth input frames 22 by generating an input frame 22 that corresponds to the processing target frame 20 and has an input pixel count equal to or greater than the initial pixel count, based on each processing target frame 20. In this embodiment, each input frame 22 has an input pixel count greater than the initial pixel count. That is, in this embodiment, each input frame 22 is an enlarged image of the processing target frame 20 corresponding to the input frame 22.
[0037] Specifically, the input frame acquisition unit 410 obtains the pixel value at the position corresponding to each pixel before the change in the processing target frame 20 by interpolation based on the change information and each pixel of each processing target frame 20, and generates each input frame 22. FIG. 6 is a diagram for explaining the processing in the input frame acquisition unit 410. In FIG. 6, the case of obtaining the n-th input frame 22_n is illustrated. For example, as shown in FIG. 6, if the pixel center of a certain pixel in the input frame 22_n to be obtained is P １，０ then the input frame acquisition unit 410 is at P in the processing target frame 20_n １，０ Find the pixel centers P' of the four pixels closest to ０，０ , P' １，０ , P' ０，１ , P' １，１ Based on the coordinates and pixel values of, obtain the pixel value of P １，０ by bilinear interpolation. Here, P' １，０ is at a position shifted from P １，０ by the amount of change indicated by the change information. The pixel values of the newly generated pixels by the enlargement process are also obtained in the same manner. In addition to bilinear interpolation, various known methods such as bicubic interpolation and Lanczos interpolation can be used as the interpolation method.
[0038] When rendering is executed so that the viewpoint C varies for each processing target frame 20, the amount of information in the time series increases. If each processing target frame 20 obtained in this way (hereinafter referred to as "variable processing target frame") is used for estimation, a higher-quality estimated frame 24 can be obtained.
[0039] On the other hand, if the variable processing target frame (or an image obtained by enlarging this) is directly input to the machine learning model 200, the accuracy of the estimation may decrease due to the influence of the change in the viewpoint C described above.
[0040] Therefore, in the image processing system 1, as described above, based on the variation information and each pixel of each processing target frame 20, the pixel value at the position corresponding to each pixel before variation in the processing target frame 20 is obtained by interpolation, and each input frame 22 is generated and input into the machine learning model 200. Thereby, the influence of the variation of the viewpoint C is corrected, so that it is possible to suppress the decrease in the accuracy of estimation.
[0041] [Machine learning model storage unit] The machine learning model storage unit 412 stores the machine learning model 200. Specifically, the machine learning model storage unit 412 stores the parameters of the machine learning model 200 (the number of convolutional layers, the number of nodes used in each convolutional layer, the weights of each node, etc.).
[0042] [Machine learning model] The machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n. Specifically, the machine learning model 200 is a model that estimates the nth estimated frame 24_n based on the nth input frame 22_n and the (n - 1)th auxiliary information 28_n - 1. Specifically, the machine learning model 200 is a convolutional neural network (CNN: Convolutional Neural Network). As the machine learning model 200, for example, a known model such as ResNet having a multi-layer structure with a residual connection mechanism or a so-called encoder-decoder type U-Net can be used. A model described in Non-Patent Document 1 may be used as the machine learning model 200.
[0043] The machine learning model 200 is a model learned by a plurality of training data including a learning input frame having an input pixel number and a learning estimated frame having an estimated pixel number. For the learning of the machine learning model 200, various known methods such as the error backpropagation method can be used.
[0044] Specifically, the machine learning model 200 includes a cumulative feature information output layer 202, an estimated frame output layer 204, and a convolutional layer 206 (see FIG. 2).
[0045] The cumulative feature information output layer 202 receives the nth input frame 22_n and the nth-1st auxiliary information 28_n-1 based on the nth-1st cumulative feature information 26_n-1 that shows the features of the first to nth-1st input frames 22, and outputs the nth cumulative feature information 26_n that shows the features of the first to nth input frames 22_n. The cumulative feature information output layer 202 may be composed of, for example, one or more convolutional layers. The cumulative feature information 26_n-1 is image information (information in bitmap format) having the same number of pixels as the number of input pixels. The cumulative feature information 26_n-1 can also be said to be a feature map that shows the features of the first to nth-1st input frames 22.
[0046] The cumulative feature information output layer 202 receives the first input frame 22_1 and the given auxiliary information, and outputs the first cumulative feature information 26_1. When n=1, since there is no previous cumulative feature information 26 and auxiliary information 28, the pre-prepared given auxiliary information is input to the cumulative feature information output layer 202 along with the first input frame 22_1.
[0047] The estimated frame output layer 204 receives the nth cumulative feature information 26_n as input and outputs the nth estimated frame 24_n. The estimated frame output layer 204 may be composed of, for example, one or more convolutional layers, similar to the cumulative feature information output layer 202. Alternatively, the estimated frame output layer 204 may be composed of one or more transposed convolutional layers (inverse convolutional layers).
[0048] The convolutional layer 206 is a layer that reduces the number of channels in the cumulative feature information 26 while maintaining the number of pixels. The cumulative feature information 26 output from the convolutional layer 206 is used for processing in the auxiliary information acquisition unit 416. The convolutional layer 206 reduces the dimension of the cumulative feature information 26, thus reducing computational costs. The convolutional layer 206 is, for example, a convolutional layer with a kernel size of 1x1, but is not limited to this.
[0049] [Estimated Frame Acquisition Unit] The estimated frame acquisition unit 414 inputs each input frame 22 to the machine learning model 200 and acquires the first to nth estimated frames 24, each having an estimated number of pixels greater than or equal to the input number of pixels and greater than or equal to the initial number of pixels. In this embodiment, the estimated frame 24 has the same number of estimated pixels as the input number of pixels. More specifically, the estimated frame acquisition unit 414 inputs the nth input frame 22_n and the n-1 auxiliary information 28_n-1 to the machine learning model 200 and acquires the nth estimated frame 24_n.
[0050] [Motion Vector Acquisition Unit] The motion vector acquisition unit 418 acquires the n-th motion vector MV, which is information indicating the amount and direction of movement from the n-th processing target frame 20_n-1 to the nth processing target frame 20_n. The motion vector MV is also called a motion vector. The motion vector MV is relative motion information that indicates the relative movement of the viewpoint C and the object O on the screen plane accompanying the movement from input frame 22_n-1 to input frame 22_n.
[0051] [Depth Information Acquisition Unit] The depth information acquisition unit 420 acquires the n-1st depth information, which indicates the depth of each pixel in the n-1st processing target frame 20_n-1, and the nth depth information, which indicates the depth of each pixel in the nth processing target frame 20_n. The depth information is also called a depth buffer, Z buffer, or depth buffer.
[0052] [Appearing Pixel Identification Unit] The appearing pixel identification unit 422 identifies the nth appearing pixel 222_n, which is a pixel in the nth input frame 22_n that displays all or part of the game object O that is not displayed in the nth input frame 22_n-1, based on the (n-1)th depth information and the nth depth information (see Figure 3). Specifically, the appearing pixel identification unit 422 identifies the nth appearing pixel 222_n based on the difference between the (n-1)th depth information and the nth depth information. Alternatively, the appearing pixel identification unit 422 may identify the nth appearing pixel 222_n based on the (n-1)th perspective projection matrix related to the nth input frame 22_n-1 and the nth perspective projection matrix related to the nth input frame 22_n. Furthermore, the appearing pixel identification unit 422 may identify the nth appearing pixel 222_n using the (n-1)th motion vector. More specifically, the occurrence pixel identification unit 422 identifies the nth occurrence pixel and generates the nth occurrence pixel value, which is image information indicating the position of the nth occurrence pixel 222_n.
[0053] [Auxiliary Information Acquisition Unit] The auxiliary information acquisition unit 416 acquires the n-th auxiliary information 28_n-1 by applying motion compensation to the n-th cumulative feature information 26_n-1 based on the n-th motion vector. Motion compensation refers to the process of moving the pixel at position x in the n-th cumulative feature information 26_n to position x' when, for example, a pixel at position x in the n-th input frame 22_n-1 has moved to position x' in the nth input frame 22_n. That is, the auxiliary information acquisition unit 416 acquires the n-th auxiliary information 28_n-1 by setting the pixel value of one or more pixels in the n-th cumulative feature information 26_n-1 to the pixel at the position where the pixel has moved according to the amount and direction of the pixel's movement, based on the n-th motion vector.
[0054] If there is movement of the game object O between the nth processing target frame 20_n and the (n-1)th processing target frame 20_n-1, and the nth estimated frame 24_n is obtained by directly inputting the nth input frame 22_n and the (n-1)th cumulative feature information 26_n-1 into the machine learning model 200, a ghosting phenomenon may occur in the output nth estimated frame 24_n, where the afterimage of the game object O that was displayed in the nth input frame 22_n is displayed.
[0055] Therefore, in the image processing system 1, motion compensation is applied to the n-th cumulative feature information 26_n-1 based on the n-th motion vector, as described above, to obtain the n-th auxiliary information 28_n-1, and when obtaining the nth estimated frame 24_n, this n-th auxiliary information 28_n-1 is input to the machine learning model 200. This makes it possible to suppress the ghosting phenomenon described above.
[0056] [4. Inspection System] As shown in Figure 1, the inspection system 100 may have a hardware configuration similar to that of the image processing system 1. That is, the inspection system 100 may include a control unit 10, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an audio output unit 19. Note that some of these units may be common with the units of the image processing system 1. Furthermore, the inspection system 100 is not limited to one computer, but may be composed of multiple computers.
[0057] Figure 7 shows an overview of the back-projection frame generation process in the inspection system. Figures 8A and 8B show the procedure for inspecting the accuracy of viewpoint information in the inspection system. In Figure 7, for the sake of simplicity, an example is shown where the game object O (hereinafter simply referred to as object O) itself does not move. Figure 8A shows an example where the viewpoint information is accurate and object O itself does not move in the virtual space, while Figure 8B shows an example where the viewpoint information is accurate and object O itself does move in the virtual space.
[0058] Figure 9 is a functional block diagram showing an example of the functions implemented in the inspection system 100. As shown in Figure 9, the inspection system 100 implements a rendering information acquisition unit 101, a back projection unit 102, a coordinate system transformation unit 103, a viewpoint transformation unit 104, a reprojection unit 105, a viewpoint movement information acquisition unit 106, an object movement information acquisition unit 107, and a display control unit 108. Each of these units is mainly implemented by the control unit 10.
[0059] The inspection system 100 is a computer for inspecting the accuracy of viewpoint information related to viewpoint C. The viewpoint information includes the orientation, position, depth range, and projection matrix for converting points in three-dimensional space to a screen plane. The viewpoint information is used to accurately identify appearing pixels (disocclusion regions) and contributes to the generation of highly accurate estimated frames 24. However, if there are errors in the viewpoint information, and this is input into the machine learning model 200, ghosting and other phenomena may occur in the estimated frames 24.
[0060] Furthermore, even if ghosting or similar phenomena occur, it is difficult to determine whether the cause is an error in the viewpoint information or an error in other input parameters such as the motion vector (MV) or depth information.
[0061] Therefore, in this embodiment, information regarding the movement of object O itself is obtained by removing the movement of the viewpoint from the motion vector MV, and an image based on that information is displayed on the display unit 18. Specifically, object movement information showing the movement of object O in virtual space is generated based on the motion vector MV, which shows the relative movement between the viewpoint and object O, and viewpoint movement information, which shows the movement of object O on the screen plane due to the movement of the viewpoint, and an object movement image based on the object movement information is displayed on the display unit 18. This is because if the object movement image based on the object movement information, which is information from which viewpoint information has been removed, is displayed as intended, it can be confirmed that there was no error in the viewpoint information.
[0062] First, the inspection system 100 converts the position of object O in input frame 22_n (current frame) to the position as seen from viewpoint C2 in input frame 22_n-1 (past frame), and generates a reprojected input frame 29 by reprojecting it onto the screen plane Sn-1. The process of generating the reprojected input frame 29 will be described below.
[0063] The rendering information acquisition unit 101 acquires color information, motion vector MV, and depth information contained in input frames 22_n-1 and 22_n, respectively, which are stored in the rendering information storage unit 404 of the image processing system 1.
[0064] Furthermore, the rendering information acquisition unit 101 acquires viewpoint information 50_n-1 corresponding to input frame 22_n-1 and viewpoint information 50_n corresponding to input frame 22_n.
[0065] The back projection unit 102 backprojects object O from the screen plane Sn into virtual space based on viewpoint information 50_n, for example, using perspective (see (a) in Figure 7). As a result, the back projection unit 102 obtains camera coordinate system position information (viewpoint coordinate system position information) 60_n of object O in the nth camera coordinate system (viewpoint coordinate system) with viewpoint C1 as the origin in the viewpoint information 50_n (see (b) in Figure 7).
[0066] The coordinate system transformation unit 103 converts the camera coordinate system position information 60_n into world coordinate system position information 70_n (see (c) in Figure 7).
[0067] The viewpoint conversion unit 104 converts the position information 70_n into camera coordinate system position information 60_n-1 based on the viewpoint information 50_n-1 (see (d) in Figure 7). The camera coordinate system position information 60_n-1 may be the position information in the camera coordinate system as seen from the viewpoint corresponding to the estimated frame 24_n-1.
[0068] The reprojection unit 105 generates a reprojection input frame 29 by projecting the object O onto the screen plane Sn-1 based on the camera coordinate system position information 60_n-1 (see (e) in Figure 7).
[0069] Through the above process, it is possible to generate a reprojected input frame 29 from the viewpoint C2 of input frame 22_n, where object O in input frame 22_n is viewed.
[0070] Next, the viewpoint motion information acquisition unit 106 acquires viewpoint motion information 80 that shows the movement of object O on the screen plane due to the movement of viewpoint C, based on the input frame 22_n and the reprojection input frame 29.
[0071] Next, the object motion information acquisition unit 107 acquires object motion information 90, which indicates the movement of object O in the virtual space, based on the viewpoint motion information 80 and the motion vector MV. The object motion information 90 is information that indicates the movement of object O itself in the virtual space, independent of the movement of viewpoint C.
[0072] The display control unit 108 then displays the object motion image OP based on the object motion information 90 on the display unit 18. The object motion information 90 is information for each pixel, and for example, pixels where the object motion information 90 is greater than a predetermined threshold may be displayed in white, and pixels where it is less than or equal to the predetermined threshold may be displayed in black. More specifically, pixels on the screen plane where at least one of the x component and y component of the object motion information 90 is greater than a predetermined threshold may be displayed in white, and pixels where both are less than or equal to the predetermined threshold may be displayed in black. In this way, the object motion image OP may be displayed on the display unit 18 as a binarized image. Note that the object motion image OP is not limited to a binarized image; it is sufficient if at least the pixels indicating the moving object O are highlighted.
[0073] If the viewpoint information 50 is correct, the object motion image OP will be displayed as intended. Specifically, white will be visible in the pixels indicating the moving object O. On the other hand, if there is an error in the viewpoint information 50, the image will appear predominantly white. This is because the viewpoint information 50 is a parameter that affects the entire image.
[0074] Figure 8A shows an example where the viewpoint information 50 is accurate and the object O itself is not moving; therefore, the object motion image OP is displayed entirely in black. Figure 8B shows an example where the viewpoint information 50 is accurate and the object O itself is moving; therefore, the pixels representing object O in the object motion image OP are displayed in white.
[0075] By employing the inspection system 100 of this embodiment, for example, a game developer can visually confirm the accuracy of the viewpoint information 50 by checking the object movement image OP displayed on the display unit 18.
[0076] In this embodiment, the movement from input frame 22_n-1 to input frame 22_n was used as an example for explanation, but similar processing may be performed between other input frames 22. Also, in this embodiment, input frame 22 was used as an example for explanation, but similar processing may be performed based on the processing target frame 20. This is because the motion vector MV and viewpoint information are the same for the processing target frame 20 and the input frame 22.
[0077] Furthermore, although this embodiment describes an example in which auxiliary information 28_n-1 is generated based on the n-1st cumulative feature information 26_n-1 which shows the features of the first to n-1st input frames 22, the embodiment is not limited to this, and the auxiliary information 28_n-1 may be generated based on feature information which shows the features of at least the n-1st input frame 22_n-1.
[0078] Furthermore, in this embodiment, the example is given where the number of input pixels is greater than the initial number of pixels, and the number of input pixels and the estimated number of pixels are the same. However, the number of input pixels and the initial number of pixels may be the same, and the estimated number of pixels may be greater than the number of input pixels. In other words, the input frame 22 does not necessarily have to be an enlarged version of the frame 20 to be processed.
[0079] Here, we will explain an example of a calculation formula for generating an object motion image (OP). Formula (1) below shows that the absolute value of the difference between the viewpoint motion information 80 (camera_movement, see formula (2) below) and the motion vector MV (motion_vector) on the screen plane is calculated, converted to the scale of the image space (0 to width, height), and compared with a threshold of 0.5. In other words, in the object motion image (OP), pixels where the difference between the viewpoint motion information 80 and the motion vector MV exceeds 0.5 pixels are highlighted, and pixels below that value are hidden. This comparison is applied independently to the x and y components of the image.
[0080]
[0081]
[0082] The above formula (2) calculates the viewpoint movement information 80, which represents the difference between the pixel position in the current frame and the pixel position in the reprojection input frame 29. The pixel coordinates (prev_pixel_pos) of the reprojection input frame 29 can be calculated by reprojection using the following formula (3). The following formula (3) represents the process of reprojecting to the coordinate system of past frames by applying various transformation matrices.
[0083]
[0084] viewZ*prevProjMat represents a transformation matrix for projecting the camera coordinate system of past frames onto a two-dimensional screen plane. prevViewMat represents a transformation matrix for converting the position information in the world coordinate system of the current frame to the position information in the camera coordinate system of past frames. invCurrViewMat represents a transformation matrix for converting the position information in the camera coordinate system of the current frame to the world coordinate system. invCurrProjMat*invViewZ represents a transformation matrix for converting the pixel coordinate system in the current frame to the camera coordinate system. curr_pixel_pos represents the pixel coordinates in the current frame.
[0085] By using the pixel coordinates (prev_pixel_pos) of the reprojection input frame 29 obtained by the above formula (3), the viewpoint movement information 80 can be calculated by the above formula (2). Furthermore, based on the above formula (1), object movement information 90 at each pixel can be calculated, and an object movement image OP can be generated.
[0086] [5. Processing performed by the inspection system] Figure 10 is a flowchart showing an example of the processing flow performed by the inspection system 100. The processing shown in Figure 10 is performed by the control unit 10 operating according to the program stored in the storage unit 12.
[0087] First, the control unit 10 acquires the color information, motion vector MV, and depth information contained in input frames 22_n-1 and 22_n, respectively, which are stored in the rendering information acquisition unit 101 of the image processing system 1 (S101). The control unit 10 also acquires viewpoint information 50_n-1 corresponding to input frame 22_n-1 and viewpoint information 50_n corresponding to input frame 22_n (S102).
[0088] Next, the control unit 10 obtains camera coordinate system position information 60_n by back-projecting the input frame 22_n onto a virtual three-dimensional space based on the viewpoint information 50_n (S103). The control unit 10 also converts the camera coordinate system position information 60_n into position information 70_n in the world coordinate system (S104).
[0089] Furthermore, the control unit 10 converts the position information 70_n into camera coordinate system position information 60_n-1 (S105). Also, based on the camera coordinate system position information 60_n-1, the control unit 10 obtains a reprojection input frame 29 by projecting the object O onto the screen plane (S106).
[0090] Furthermore, the control unit 10 acquires viewpoint motion information 80 that indicates the movement of object O on the screen plane due to the movement of viewpoint C, based on the input frame 22_n and the reprojection input frame 29 (S107).
[0091] Furthermore, the control unit 10 obtains object motion information 90, which indicates the movement of object O in the virtual space, by subtracting viewpoint motion information 80 from the motion vector MV (S108). Then, the control unit 10 displays an object motion image OP based on the object motion information 90 on the display unit 18 (S109).
Claims
1. An image processing system that obtains first to nth (n is a natural number of 2 or more) input frames by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint in a virtual space, and inputs the nth (n=2, 3, ..., N) input frame, the nth-1st feature information showing the features of the nth-1st input frame, and the nth-1st auxiliary information based on viewpoint information relating to the viewpoint into a machine learning model to obtain the nth estimated frame output from the machine learning model, wherein the at least one processor obtains the nth-1st viewpoint information corresponding to the nth-1st input frame, obtains the nth viewpoint information corresponding to the nth input frame, obtains relative motion information showing the relative movement of the viewpoint and the object on the screen plane accompanying the movement from the nth-1st input frame to the nth input frame, and obtains the nth viewpoint coordinate system position information of the object in the nth viewpoint coordinate system with the viewpoint in the nth viewpoint information as the origin, based on the nth viewpoint information. An inspection system that converts the nth viewpoint coordinate system position information into the nth position information in the virtual space; converts the nth position information into the nth-1 viewpoint coordinate system position information in the nth-1 viewpoint coordinate system with the viewpoint in the nth-1 viewpoint information as the origin, based on the nth-1 viewpoint information; obtains a reprojection input frame by projecting the object onto the screen plane based on the nth viewpoint coordinate system position information; obtains viewpoint motion information showing the movement of the object on the screen plane due to the movement of the viewpoint, based on the nth input frame and the reprojection input frame; obtains object motion information showing the movement of the object in the virtual space, based on the relative motion information and the viewpoint motion information; and displays an object motion image based on the object motion information on the display unit.
2. The inspection system according to claim 1, wherein at least one processor highlights pixels in the object motion image in which the object motion information is greater than a predetermined threshold.
3. An inspection method for inspecting the accuracy of viewpoint information in an image processing system that obtains first to nth (n is a natural number of 2 or more) input frames by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint in a virtual space, and inputs the nth (n=2, 3, ..., N) input frame, the nth-1st feature information showing the characteristics of the nth-1st input frame, and the nth-1st auxiliary information based on viewpoint information relating to the viewpoint into a machine learning model, thereby obtaining the nth estimated frame output from the machine learning model, wherein the processor obtains the nth-1st viewpoint information corresponding to the nth-1st input frame, obtains the nth viewpoint information corresponding to the nth input frame, obtains relative motion information showing the relative movement of the viewpoint and the object on the screen plane accompanying the movement from the nth-1st input frame to the nth input frame, and obtains the nth viewpoint coordinate system position information of the object in the nth viewpoint coordinate system with the viewpoint in the nth viewpoint information as the origin, based on the nth viewpoint information. An inspection method comprising: converting the nth viewpoint coordinate system position information into the nth position information in the virtual space; converting the nth position information into the nth-1 viewpoint coordinate system position information in the nth-1 viewpoint coordinate system with the viewpoint in the nth-1 viewpoint information as the origin, based on the nth-1 viewpoint information; obtaining a reprojection input frame by projecting the object onto the screen plane based on the nth viewpoint coordinate system position information; obtaining viewpoint motion information showing the movement of the object on the screen plane due to the movement of the viewpoint, based on the nth input frame and the reprojection input frame; obtaining object motion information showing the movement of the object in the virtual space, based on the relative motion information and the viewpoint motion information; and displaying an object motion image based on the object motion information on the display unit.
4. An image processing system that obtains first to nth (n is a natural number of 2 or more) input frames by rendering three-dimensional data showing one or more objects viewed from a predetermined viewpoint in a virtual space, and inputs the nth (n=2, 3, ..., N) input frame, the nth-1st feature information showing the characteristics of the nth-1st input frame, and the nth-1st auxiliary information based on viewpoint information relating to the viewpoint into a machine learning model to obtain the nth estimated frame output from the machine learning model, the program for checking the accuracy of the viewpoint information, comprising: means for obtaining the nth-1st viewpoint information corresponding to the nth-1st input frame; means for obtaining the nth viewpoint information corresponding to the nth input frame; means for obtaining relative motion information showing the relative movement of the viewpoint and the object on the screen plane accompanying the movement from the nth-1st input frame to the nth input frame; means for obtaining the nth viewpoint coordinate system position information of the object in the nth viewpoint coordinate system with the viewpoint in the nth viewpoint information as the origin, based on the nth viewpoint information. A program for causing a computer to function as follows: means for converting the nth viewpoint coordinate system position information into the nth position information in the virtual space; means for converting the nth position information into the nth-1 viewpoint coordinate system position information in the nth-1 viewpoint coordinate system with the viewpoint in the nth-1 viewpoint information as the origin, based on the n-1 viewpoint information; means for obtaining a reprojection input frame by projecting the object onto a screen plane based on the n-1 viewpoint coordinate system position information; means for obtaining viewpoint movement information indicating the movement of the object on the screen plane due to the movement of the viewpoint, based on the nth input frame and the reprojection input frame; means for obtaining object movement information indicating the movement of the object in the virtual space, based on the relative movement information and the viewpoint movement information; and means for displaying an object movement image based on the object movement information on a display unit.