Image processing system, image processing method, and program
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SONY INTERACTIVE ENTERTAINMENT LLC
- Filing Date
- 2025-02-03
- Publication Date
- 2026-08-06
Smart Images
Figure JP2025003472_06082026_PF_FP_ABST
Abstract
Description
Image processing system, image processing method, and program
[0001] This invention relates to an image processing system, an image processing method, and a program.
[0002] Conventionally, a technique (super-resolution) has been known that uses machine learning models to estimate high-resolution still images based on low-resolution still images (see Non-Patent Document 1 below).
[0003] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014
[0004] The inventors of this application are considering using a machine learning model with a recursive structure to achieve super-resolution of moving images such as game screens (see Japanese Patent Application No. 2023-104102). In estimating high-resolution frames using such a machine learning model with a recursive structure, not only information from the current frame to be processed but also information from past frames of that frame contributes. By contributing information from past frames in addition to the current frame to the estimation, it is expected that the estimation accuracy of the machine learning model will improve. In moving images, there is usually movement between past frames and the current frame, and if the information from past frames is input directly into the machine learning model, a ghosting phenomenon may occur in the estimated image output by the machine learning model, where afterimages are displayed. To prevent this ghosting phenomenon, pixels included in the information from past frames that have moved to different pixel positions in the frame being processed should be aligned to their corresponding pixel positions in the frame being processed using motion compensation with motion vectors.
[0005] Incidentally, past frame information may contain pixels whose contribution to estimation should be adjusted (hereinafter sometimes referred to as "adjustment target pixels"). One example is when an object is displayed in either the past frame or the current frame but not in the other. Another example is when, for a certain pixel, there is movement between the past frame and the current frame, but no motion vector exists for that pixel, making the aforementioned motion compensation for alignment impossible.
[0006] One possible method for adjusting the contribution of the target pixels to the estimation is to prepare a contribution adjustment mask that indicates the degree of contribution of each pixel in the past frame's information to the estimation, and then input the past frame's information to which the contribution adjustment mask has been applied into a machine learning model. By doing so, it is thought that the contribution of the target pixels in the past frame's information will be suppressed in the estimation performed by the machine learning model.
[0007] However, manually creating such contribution adjustment masks is extremely time-consuming. In particular, there are many scenes in game footage where it is desirable to use contribution adjustment masks. For these scenes, manually identifying the pixels to be adjusted and setting the values of each pixel in the contribution adjustment mask for each target pixel to appropriate values is extremely time-consuming.
[0008] The present invention has been made in view of the above problems, and one of its objectives is to provide an image processing system, an image processing method, and a program that enable the creation of a contribution adjustment mask with minimal effort.
[0009] An image processing system according to one embodiment of the present invention is an image processing system comprising at least one memory for storing one or more programs and at least one processor, wherein the at least one processor sequentially generates processing target frames for times 1 to N (where N is a natural number of 2 or more) having the initial number of pixels, based on a plurality of types of drawing basic information maps for times 1 to N (where N is a natural number of 2 or more), each having a value for determining the color of each of a plurality of drawing target pixels, each of which is arranged in a virtual space and each of which is represented by three-dimensional data, according to the one or more programs; an input frame acquisition step of acquiring an input frame for time n having an input pixel count equal to or greater than the initial number of pixels, based on the processing target frame for time n (where n is a natural number of 2 or more and N or less); a past information map acquisition step of acquiring a past information map for time n-1 having the input pixel count, based on the input frame for at least time n-1; and the past information map for time n-1 The process involves: an auxiliary information map acquisition step to acquire an auxiliary information map at time n-1 which is input to the machine learning model along with the input frame at time n, by applying a contribution adjustment mask at time n that indicates how much the value of each pixel in the information map contributes to the estimation by the machine learning model; an estimation step to acquire an estimated frame at time n based on the input frame at time n, the auxiliary information map at time n-1, and the machine learning model; and the at least one processor further executes an argument map acquisition step to acquire at least one of the plurality of basic drawing information maps at time n as an argument map at time n, according to one or more programs, a function value map acquisition step to acquire a function value map at time n which has the function value for each pixel in the argument map at time n and the number of input pixels, and in the auxiliary information map acquisition step, the function value map at time n is used as the contribution adjustment mask at time n-1.
[0010] This figure shows an example of the hardware configuration of the image processing system according to this embodiment. This is a functional block diagram showing an example of the functions realized by the image processing system according to this embodiment. This figure shows an example of the processing of the rendering unit. This figure shows an example of the processing of the estimation system. This figure explains the process of obtaining the contribution adjustment mask. This figure explains the process of obtaining the contribution adjustment mask. This figure shows an example of the processing of the rendering program according to this embodiment. This figure shows an example of the processing of the estimation program according to this embodiment. This figure explains the process of obtaining the contribution adjustment mask in a modified example.
[0011] Embodiments of the present invention will be described below with reference to the drawings.
[0012] [1. Hardware Configuration of Image Processing System 1] Figure 1 is a diagram showing an example of the hardware configuration of the image processing system 1 according to this embodiment. The image processing system 1 is a computer such as a game console (game machine). As shown in Figure 1, the image processing system 1 includes a control unit 10, a storage unit 12, a display unit 14, a communication unit 16, an operation unit 18, and an audio output unit 19.
[0013] The control unit 10 includes a program control device such as a CPU that operates according to a program installed in the image processing system 1, for example. The control unit 10 also includes a GPU (Graphics Processing Unit) that draws images to a frame buffer based on graphics commands and data supplied from the CPU.
[0014] The memory unit 12 includes, for example, a main memory such as ROM or RAM, and an auxiliary memory such as an HDD or SSD. The memory unit 12 stores programs executed by the control unit 10. The memory unit 12 stores, for example, a game program (game software). The game program includes a rendering program, which will be described later. The memory unit 12 also stores an estimation program, which will be described later. Furthermore, the memory unit 12 has an area reserved for a frame buffer where images are drawn by the GPU.
[0015] The display unit 14 is a display device such as a liquid crystal display or an organic EL display, and displays various images according to the instructions of the control unit 10.
[0016] The communication unit 16 is, for example, a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.
[0017] The operation unit 18 is a user interface such as a keyboard, mouse, or game console controller, and receives user input and outputs a signal indicating the content of that input to the control unit 10.
[0018] The audio output unit 19 is, for example, a speaker, and outputs the audio indicated by the audio data generated by the image processing system 1.
[0019] In addition to the devices described above, the image processing system 1 may also include an optical disc drive for reading optical discs such as DVD-ROMs and Blu-ray® discs, and a USB (Universal Serial Bus) port.
[0020] [2. Functional Configuration of Image Processing System 1] The functions realized by the image processing system 1 according to this embodiment will be described below with reference to Figure 2. Figure 2 is a functional block diagram showing an example of the functions realized by the image processing system 1 according to this embodiment. As shown in Figure 2, the image processing system 1 includes a rendering system 20 and an estimation system 22. The rendering system 20 is realized by the control unit 10 executing a rendering program stored in the storage unit 12. The estimation program is realized by the control unit 10 executing an estimation program stored in the storage unit 12.
[0021] [2-1. Rendering System 20] As shown in Figure 2, the rendering system 20 includes a game processing unit 200 and a rendering unit 202. The game processing unit 200 and the rendering unit 202 are mainly implemented by a control unit 10 and a storage unit 12.
[0022] [2-1-1. Game Processing Unit 200] The game processing unit 200 performs various processes related to the game. For example, the game processing unit 200 performs processes such as placing objects O in a virtual three-dimensional space, operating or moving objects O, and changing the viewpoint from which the virtual three-dimensional space is viewed, in response to the game program executed by the control unit 10 or user input received by the operation unit 18. Objects O are composed of primitives such as polygons represented by three-dimensional data. The three-dimensional data includes geometric information indicating the positions of vertices, phase information indicating how the vertices are connected, and attribute information such as color.
[0023] [2-1-2. Rendering Unit 202] The rendering unit 202 sequentially generates processing frames for times 1 to N (where N is a natural number greater than or equal to 2) with an initial number of pixels by drawing (rendering) each of a plurality of objects O that are arranged in a virtual three-dimensional space and each represented by three-dimensional data. The initial number of pixels is, for example, 1920 × 1080 (1080p), but is of course not limited to this. As shown in Figure 2, the rendering unit 202 includes a drawing target pixel determination unit 2020, a drawing basic information map acquisition unit 2022, and a processing target frame generation unit 2024. The rendering unit 202 will be described below with further reference to Figure 3. Figure 3 is a diagram showing an example of the processing of the rendering unit 202.
[0024] [2-1-2-1. Drawing Target Pixel Determination Unit 2020] The drawing target pixel determination unit 2020 determines multiple drawing target pixels TP at time m (where n is a natural number between 1 and N). ｍ Determine the multiple pixels to be drawn TP ｎ In this, multiple objects O, each represented by 3D data and placed in a virtual space, are drawn (Figure 3). Specifically, the drawing target pixel determination unit 2020 maps the shape determined by the vertex coordinates of each object O to multiple pixels, and the multiple pixels are used to determine multiple drawing target pixels TP ｍIt is determined as such. The vertex coordinates of the object O are obtained by converting the vertex coordinates in the world coordinate system (the coordinate system of the three-dimensional virtual space) into coordinates in the two-dimensional screen coordinate system through view transformation and projection transformation. The processing of the drawing target pixel determination unit 2020 can also be said to be a part of the rasterization processing.
[0025] [2-1-2-2. Drawing basic information map acquisition unit 2022] The drawing basic information map acquisition unit 2022 acquires a plurality of types of drawing basic information maps RM at time m based on the three-dimensional data representing each object O (FIG. 3). ｍ Each is acquired (FIG. 3). The plurality of types of drawing basic information maps RM ｍ Each has values for determining the colors of a plurality of drawing target pixels TP ｍ for each pixel and has an initial number of pixels. Specifically, for each drawing target pixel TP ｍ , the drawing basic information map acquisition unit 2022 calculates, by interpolation, a value (attribute value) for determining the color of the drawing target pixel TP ｍ from the attribute values associated with the vertex coordinates of each object O. The values for determining the colors of the plurality of drawing target pixels TP ｍ each include, for example, the RGB values of the base color, depth, normal vector, transparency, material ID, etc. That is, the plurality of types of drawing basic information maps RM ｍ include a base color map, a depth map, a normal vector map, a transparency map, a material ID map, etc. The processing of the drawing basic information map acquisition unit 2022 can also be said to be a part of the rasterization processing.
[0026] The drawing basic information map acquisition unit 2022 stores the acquired plurality of types of drawing basic information maps RM at time m ｍ in the rendering information storage unit 24. Thereby, the drawing basic information map acquisition unit 2022 provides the plurality of types of drawing basic information maps RM at time m ｍ to the estimation program. Note that the rendering information storage unit 24 is mainly realized by the storage unit 12.
[0027] [2-1-2-3. Processing target frame generation unit 2024] The processing target frame generation unit 2024 generates a plurality of types of drawing basic information maps RM at time mｍ Based on this, frame f at time m is the frame to be processed. ｍ The process generates the following (Figure 3). The processing target frame generation unit 2024 generates, for example, multiple types of drawing basic information maps RM for time m. ｍ Based on the information of the light source placed in the virtual 3D space, each pixel TP to be drawn ｍ By performing shading calculations, the target pixel TP ｍ The color is tentatively determined. The processing target frame generation unit 2024 generates each drawing target pixel TP whose color has been tentatively determined. ｍ Depth tests, alpha tests, stencil tests, etc. are performed on the target pixel TP for rendering. ｍ The color may be determined. The processing target frame generation unit 2024 then determines the drawing target pixel TP ｍ The color obtained by applying post-effects such as tone mapping, bloom, and depth of field to the color of frame f is the target frame for processing. ｍ The color of each pixel may be obtained. Alternatively, the processing target frame generation unit 2024 may obtain the determined drawing target pixel TP ｍ The color itself is the target frame f ｍ The color of each pixel may also be obtained.
[0028] The processing target frame generation unit 2024 generates the processing target frame f at the acquired time m. ｍ This is stored in the rendering information storage unit 24. As a result, the processing target frame generation unit 2024 generates the processing target frame f at time m. ｍ This is provided to the estimation program.
[0029] [2-2. Estimation System 22] As shown in Figure 2, the estimation system 22 includes an input frame acquisition unit 220, an estimation unit 221, a machine learning model storage unit 222, an auxiliary information map acquisition unit 223, an argument map acquisition unit 224, and a function value map acquisition unit 225. The input frame acquisition unit 220, the estimation unit 221, the auxiliary information map acquisition unit 223, the argument map acquisition unit 224, and the function value map acquisition unit 225 are mainly implemented by the control unit 10. The machine learning model storage unit 222 is mainly implemented by the storage unit 12. The processing of the estimation system 22 will be described below with further reference to Figure 4. Figure 4 is a diagram showing an example of the processing of the estimation system 22.
[0030] [2-2-1. Input Frame Acquisition Unit 220] The input frame acquisition unit 220 processes the frame f at time n (where n is a natural number between 2 and N). ｎ Based on this, input frame F at time n has a number of input pixels greater than the initial number of pixels. ｎ The input pixels are, for example, 3840 x 2160 (4K), but are not limited to this. Specifically, the input frame acquisition unit 220 refers to the rendering information storage unit 24 and obtains the frame f to be processed at time n. ｎ The input frame acquisition unit 220 then obtains the frame f to be processed at the acquired time n. ｎ The input frame F at time n is subjected to scaling and interpolation processing. ｎ It generates the following. If the rendering unit 202 applies viewpoint fluctuations (jitter) during rendering, the input frame acquisition unit 220 obtains the fluctuation information and each processing target frame f ｎ Based on each pixel and the frame f to be processed, ｎ The pixel values at the positions corresponding to each pixel before the change are interpolated, and each input frame F ｎ It may be generated (see Japanese Patent Application No. 2023-104102).
[0031] Here, input frame F ｎ The frame to be processed is f ｎIt should be noted that even if a frame has a higher pixel count than the reference frame, its image quality is not necessarily sufficiently improved. In other words, frame quality is not simply about the number of pixels (high resolution). Frame quality may be evaluated based on factors such as the signal-to-noise ratio, spatial frequency reproduction accuracy, and temporal stability (minimal artifacts and flickering when multiple frames are displayed consecutively), either individually or as a combination of these factors, compared to a reference frame.
[0032] [2-2-2. Estimation Unit 221] The estimation unit 221 receives the input frame F at time n. ｎ And, the auxiliary information map AM for time n-1 ｎ-1 Based on the machine learning model M, an estimated frame F at time n has an estimated number of pixels. ｎ 'and, cumulative feature information map FM at time n ｎ And, is obtained (Figure 4). In this embodiment, the case where the estimated number of pixels is 4K, the same as the input number of pixels, is shown as an example, but the estimated number of pixels can be any number greater than or equal to the input number of pixels and greater than or equal to the initial number of pixels. The estimation unit 221 reads the parameters of the machine learning model M from the machine learning model storage unit 222 and the input frame F at time n. ｎ and the auxiliary information map AM for time n-1 ｎ-1 The and are input to a machine learning model M, and the estimated frame F of time n is output from the machine learning model M. ｎ 'and cumulative feature information map FM at time n ｎ Obtain the following: Machine learning model M, Cumulative Feature Information Map FM ｎ and auxiliary information map AM ｎ-1 Further details will be provided later.
[0033] [2-2-3. Machine Learning Model Storage Unit 222] The machine learning model storage unit 222 stores the machine learning model M. Specifically, the machine learning model storage unit 222 stores the parameters of the machine learning model M (such as the number of convolutional layers, the number of nodes used in each convolutional layer, and the weights of each node).
[0034] [Machine Learning Model M] The machine learning model M uses the input frame F at time n. ｎ And, the auxiliary information map AM for time n-1 ｎ-1The system accepts the following as inputs and estimates the frame F at time n. ｎ 'and, cumulative feature information map FM at time n ｎ This is a model that outputs the following:
[0035] The machine learning model M is, for example, a model based on a convolutional neural network (CNN). As the machine learning model M, known models such as the multilayer structure ResNet with a residual connection mechanism, or the so-called encoder-decoder type U-Net can be used. The model described in Non-Patent Document 1 may also be used as the machine learning model M.
[0036] The machine learning model M is a model trained using training data that associates training input frames with input pixel counts generated based on training target frames with initial pixel counts, with training estimated frames with estimated pixel counts. Various known methods, such as backpropagation, can be used to train the machine learning model M.
[0037] The machine learning model M specifically includes a cumulative feature information map output layer L1, an estimated frame output layer L2, and a convolutional layer L3 (Figure 4).
[0038] [Cumulative Feature Information Map Output Layer L1] The cumulative feature information map output layer L1 receives the input frame F at time n. ｎ And, the auxiliary information map AM for time n-1 ｎ-1 It accepts and as input, and generates an accumulated feature information map FM at time n relating to the features of the input frame at time n. ｎ The cumulative feature information map output layer L1 may consist of, for example, one or more convolutional layers. ｎ This is image information (information in bitmap format) with an input pixel count. Cumulative Feature Information Map FM ｎ This is the input frame F at time n. ｎ Each pixel has one or more values relating to its features. That is, cumulative feature information map FM ｎ It has one or more channels.
[0039] The cumulative feature information map output layer L1 receives the input frame at time 1 and a given auxiliary information map as input, and outputs the cumulative feature information map at time 1. When n=1, since no previous cumulative feature information maps or auxiliary information maps exist, a pre-prepared given auxiliary information map is input to the cumulative feature information map output layer L1 along with the input frame at time 1. The given auxiliary information map is data in the same format as the auxiliary information maps from time 2 to time N. That is, the given auxiliary information map has the same number of input pixels and the same number of channels as the auxiliary information maps from time 2 to time N.
[0040] [Estimated Frame Output Layer L2] The estimated frame output layer L2 is the cumulative feature information map FM at time n. ｎ The input is accepted, and the estimated frame F at time n. ｎ The output is ' (Figure 4). The estimated frame output layer L2 may be composed of, for example, one or more convolutional layers, similar to the cumulative feature information map output layer L1. Alternatively, the estimated frame output layer L2 may be composed of one or more transposed convolutional layers (inverse convolutional layers).
[0041] [Convolutional layer L3] Convolutional layer L3 is a cumulative feature information map FM ｎ This layer reduces the number of channels while maintaining the number of pixels (Figure 4). The cumulative feature information output from the convolutional layer L3 is used for processing in the auxiliary information map acquisition unit 223, which will be described later. According to the convolutional layer L3, the cumulative feature information map FM ｎ Since the dimensionality is reduced, the computational cost can be lowered. The convolutional layer L3 is, for example, a convolutional layer with a kernel size of 1x1, but is not limited to this.
[0042] [2-2-4. Auxiliary Information Map Acquisition Unit 223] Before describing the processing of the Auxiliary Information Map Acquisition Unit 223, the Auxiliary Information Map AM ｎ-1 And, auxiliary information map AM ｎ-1 This section explains the contribution adjustment mask used to obtain the following: Auxiliary information map AM at time n-1. ｎ-1 This is the cumulative feature information map FM at time n-1, relating to the features of the input frame at time n-1. ｎ-1It is obtained based on (past information map). That is, the auxiliary information map AM for time n-1. ｎ-1 This relates to the characteristics of the input frame at least at time n-1. Also, the auxiliary information map AM at time n-1. ｎ-1 This is the cumulative feature information map FM at time n-1. ｎ-1 Similarly, it has an input pixel count.
[0043] As explained earlier, in the machine learning model M having a recursive configuration used in this embodiment, the input frame F at time n ｎ Furthermore, at least the auxiliary information map AM at time n-1 relating to the characteristics of the input frame at time n-1 ｎ-1 Also, estimation (estimated frame F at time n) ｎ This contributes to the output of '. This increases the amount of information available for estimation, resulting in a high-quality estimated frame F. ｎ You can obtain '.
[0044] However, the cumulative feature information map FM at time n-1 ｎ-1 This may include pixels that should be adjusted for their contribution to the estimation by the machine learning model M, i.e., pixels to be adjusted.
[0045] One example is when an object is displayed in either a past frame or the current frame that is not displayed in the other. Specifically, this could be a case where an object that was not displayed in a past frame suddenly appears in the current frame, or an object that was displayed in a past frame suddenly disappears in the current frame. In such cases, the information from the past frame (the cumulative feature information map FM at time n-1) is used. ｎ-1 If the contribution of the pixels corresponding to the display position of the object in question to the estimation by the machine learning model is large, ghosting may occur in the estimated frame output by the machine learning model, which could actually lead to a decrease in image quality.
[0046] A second example is when, despite movement occurring between past and present frames in a given pixel region, no motion vector exists in that pixel region, making motion compensation for alignment (see Japanese Patent Application No. 2023-104102) impossible. For example, in the case of a pixel region with an animated texture (such as an object reflected in glass or a mirror, a rippling water surface, or an electronic billboard), despite movement occurring between past and present frames, no motion vector exists in that pixel region. Similarly, in the case of two-dimensional moving images such as rain or sparks superimposed on a rendered image, despite movement occurring between past and present frames, no motion vector exists in that pixel region. In such cases, information from past frames (the cumulative feature information map FM at time n-1) is also required. ｎ-1 If the contribution of the pixel region in question to the estimation by the machine learning model is large, ghosting may occur in the estimated frame output by the machine learning model, which could actually lead to a decrease in image quality.
[0047] Therefore, in this embodiment, the auxiliary information map acquisition unit 223 acquires the cumulative feature information map FM at time n-1. ｎ-1 By applying a contribution adjustment mask at time n, the auxiliary information map AM at time n-1 is obtained. ｎ-1 The method was modified to obtain the following: The contribution adjustment mask at time n is the cumulative feature information map FM at time n-1. ｎ-1 This indicates the extent to which the value of each pixel included in the image contributes to the estimation by the machine learning model M. The contribution adjustment mask at time n is the cumulative feature information map FM at time n-1. ｎ－１ This is a mask used to reduce the contribution of a target pixel in the image to be adjusted to the estimation by the machine learning model M. The contribution adjustment mask is image information with the number of input pixels. The range of possible values for each pixel in the contribution adjustment mask at time n is, for example, a real number between 0 and 1.
[0048] The contribution adjustment mask obtained in this way is used for the cumulative feature information map FM at time n-1. ｎ-1 When applied to this, in estimation by the machine learning model M, the cumulative feature information map FM at time n-1 ｎ-1The contribution of the pixels to be adjusted included therein can be suppressed.
[0049] However, creating such a contribution adjustment mask manually is extremely time-consuming. Particularly in game videos, there are many scenes where it is desirable to use a contribution adjustment mask. Manually identifying the pixels to be adjusted and setting the values of each pixel of the contribution adjustment mask for each pixel to be adjusted to appropriate values is very time-consuming for these scenes.
[0050] Therefore, in the present embodiment, although details will be described later, one of the plurality of types of drawing basic information maps RM ｎ at time n is obtained as the argument map IM ｎ at time n. Next, a predetermined function is applied to the value of each pixel included in the argument map IM ｎ at time n to calculate a function value, and a function value map OM ｎ at time n having the function value for each pixel and having the number of input pixels is obtained (FIG. 5B). In the present embodiment, this function value map OM ｎ at time n is used as the contribution adjustment mask at time n. Thereby, a contribution adjustment mask can be created with less effort.
[0051] [2-2-5. Argument Map Acquisition Unit 224] Hereinafter, the details of the argument map acquisition unit 224 and the function value map acquisition unit 225 will be described while further referring to FIGS. 5A and 5B. FIGS. 5A and 5B are diagrams for explaining the process of acquiring a contribution adjustment mask.
[0052] The argument map acquisition unit 224 acquires any one of the plurality of types of drawing basic information maps RM ｎ at time n as the argument map IM ｎ at time n. In the present embodiment, an example is given in which the argument map IM ｎ at time n has a value indicating the transparency of each object O for each pixel. That is, in the present embodiment, the argument map IM ｎ at time n is a transparency map. As an example, a processing target frame f ｎThe transparency map at time n corresponding to (FIG. 5A) is shown in the upper part of FIG. 5B as the argument map IM at time n. The argument map acquisition unit 224 refers to the rendering information storage unit 24 and acquires the argument map IM at time n. ｎ Referring to the rendering information storage unit 24, the argument map acquisition unit 224 acquires the argument map IM at time n. ｎ
[0053] [2-2-6. Function value map acquisition unit 225] The function value map acquisition unit 225 applies a predetermined function to the value of each pixel included in the argument map IM at time n to obtain a function value, and has the function value for each pixel and the input pixel number, and obtains the function value map OM at time n (FIG. 5B). In the present embodiment, the predetermined function includes a conversion formula for converting the value of each pixel included in the argument map IM at time n into a function value. The conversion formula may use, for example, the following formula 1. In formula 1, f is the function value of each pixel included in the function value map OM at time n, c is a coefficient, and x is the value of each pixel included in the argument map IM at time n. Of course, formula 1 is merely an example, and any formula may be used as the conversion formula.
[0054]
[0055] The coefficient included in the conversion formula may be preset by a game developer or the like. In this way, if the game developer or the like presets the coefficient included in the conversion formula, it becomes easier to adjust the value of each pixel of the function value map OM (that is, the contribution adjustment mask) within the range desired by the game developer or the like. In this case, the function value map acquisition unit 225 may refer to the storage unit that stores the set coefficient and acquire the function value map OM at time n.
[0056] In FIG. 5B, for the convenience of drawing, the argument map IM at time n and the function value map OM at time n are drawn in the same way, but in reality, the argument map IM at time n has the initial pixel number, while the function value map OM at time n ｎ and the function value map OM at time n ｎ are drawn in the same way, but in reality, the argument map IM at time n ｎ has the initial pixel number, while the function value map OM at time n ｎ [has the input pixel number.The input has a number of pixels. Therefore, in this embodiment, the predetermined function includes operations to perform scaling and interpolation on image data with an initial number of pixels to obtain image data with an input number of pixels. Before being processed by the function value map acquisition unit 225, the argument map IM at time n ｎ When scaling and interpolation are performed, the specified function does not need to include the scaling and interpolation operations described above.
[0057] As mentioned above, the auxiliary information map acquisition unit 223 acquires the function value map OM for time n. ｎ This is used as the contribution adjustment mask for time n. This eliminates the need to manually create the contribution adjustment mask, thus allowing it to be created with less effort.
[0058] Another way to reduce the effort required to create a contribution adjustment mask is to use an argument map IM for time n. ｎ It is also conceivable to use the same thing directly as a contribution adjustment mask. However, the pixel values of the contribution adjustment mask obtained by this method may become too large or too small, resulting in an inappropriate value. In this embodiment, multiple drawing basic information RM at time n ｎ One of them is the argument map IM for time n ｎ Obtained as, the argument map IM for time n ｎ A predetermined transformation is performed on the value of each pixel contained in the input to calculate a function value, and a function value map OM at time n, which has the function value for each pixel and the number of input pixels, is generated. ｎ Obtain the function value map OM at time n-1. ｎ This is used as the contribution adjustment mask for time n. This allows for the creation of a contribution adjustment mask with appropriate pixel values set with minimal effort.
[0059] In this embodiment, the auxiliary information map acquisition unit 223 acquires the function value map OM for time n. ｎ The input frame F at time n is based on the function value of each pixel contained within it. ｎ The value of each pixel and the cumulative feature information map FM at time n-1. ｎ-1 Based on the weighted average of the values of each pixel, the auxiliary information map AM at time n-1 ｎ-1The value of each pixel is calculated. That is, the auxiliary information map acquisition unit 223 calculates the function value map OM at time n. ｎ Based on the function value of each pixel contained in the input frame F at time n, ｎ The value of each pixel and the cumulative feature information map FM at time n-1. ｎ-1 By alpha blending the values of each pixel, the auxiliary information map AM at time n-1 is obtained. ｎ-1 The value of each pixel is calculated. More specifically in this embodiment, the auxiliary information map acquisition unit 225 acquires the input frame F at time n. ｎ Function value map OM of the value of each pixel and time n ｎ The product of the value of the pixel and the cumulative feature information map FM at time n-1 ｎ-1 The value of the corresponding pixel and the function value map OM from 1 to time n ｎ The product of the value obtained by subtracting the value of the pixel and the sum of the two values is used in the auxiliary information map AM at time n-1. ｎ-1 It is calculated as the value of the corresponding pixel.
[0060] [3. Processing performed by Image Processing System 1] The following describes the processing performed by Image Processing System 1.
[0061] [3-1. Processing of the rendering program] Figure 6 shows an example of the processing of the rendering program according to this embodiment. The processing shown in Figure 6 is performed by the control unit 10 operating in accordance with the rendering program stored in the storage unit 12.
[0062] As shown in Figure 6, the control unit 10 initializes the variable m to 1 (S600), and calculates the multiple drawing target pixels TP at time m when each of the multiple objects O is drawn. ｎ The control unit 10 determines (S602). Based on the three-dimensional data representing each object O, the control unit 10 determines multiple types of drawing basic information maps RM for time m. ｍ The control unit 10 then obtains each of the following (S604). Then, the control unit 10 obtains multiple types of drawing basic information maps RM for time m. ｍ Based on this, frame f at time m is the frame to be processed. ｍThe variable m is generated (S606). The control unit 10 determines whether the variable m is equal to N (i.e., whether processing has been performed for all processing target frames from time 1 to N) (S608). If it determines that the variable m is not equal to N (S608: No), it updates the variable m to m+1 (S610) and executes the processing from S602 to S608 again. If the control unit 10 determines that the variable m is equal to N (S608: Yes), it terminates this process.
[0063] [3-2. Processing of the Estimation Program] Figure 7 shows an example of the processing of the estimation program according to this embodiment. The processing shown in Figure 7 is performed by the control unit 10 operating in accordance with the estimation program stored in the storage unit 12.
[0064] (1) Processing at n=1 First, the control unit 10 obtains the input frame at time 1 based on the frame to be processed at time 1 (S700). Then, the control unit 10 obtains the estimated frame at time 1 and the cumulative feature information at time 1 based on the input frame at time 1, given auxiliary information, and the machine learning model M (S702).
[0065] (2) When n≧2, the processing control unit 10 initializes the variable n to 2 (S704) and processes the frame f at time n. ｎ Based on this, input frame F at time n ｎ Obtain (S706).
[0066] Next, the control unit 10 generates a map RM of multiple types of drawing basic information for time n. ｎ One of them is the argument map IM for time n ｎ It is obtained as (S708). The control unit 10 retrieves the argument map IM of time n. ｎ A predetermined function is applied to obtain the function value, and the function value map OM at time n is obtained. ｎ Obtain (S710).
[0067] The control unit 10 generates the cumulative feature information map FM at time n-1. ｎ－１ For this, the function value map OM at time n ｎ By applying this as a contribution adjustment mask for time n-1, the auxiliary information map AM for time n-1 ｎ-1The control unit 10 then obtains the input frame F at time n. ｎ And, the auxiliary information map AM for time n-1 ｎ-1 Based on the machine learning model M, the estimated frame F at time n is calculated. ｎ 'and, cumulative feature information map FM at time n ｎ The control unit 10 obtains the value of the variable n (S714). The control unit 10 determines whether the variable n is equal to N (i.e., whether processing has been performed for all processing target frames from time 2 to N) (S716). If it determines that the variable n is not equal to N (S716: No), it updates the variable n to n+1 (S718) and executes the processing in S706 to 716 again. If the control unit 10 determines that the variable n is equal to N (S716: Yes), it terminates this process.
[0068] Note that the process shown in Figure 7 is just one example and is not limited to this example. For example, the execution order of S706 and S708 to S712 may be reversed.
[0069] [4. Modifications] The present invention is not limited to the embodiments described above. Furthermore, the specific strings and numbers shown above and in the drawings are illustrative examples and the invention is not limited to these strings and numbers.
[0070] Figure 8 illustrates the process of obtaining the contribution adjustment mask in a modified example. In this embodiment, the argument map IM at time n ｎ The example given is that it is a transparency map, but it is not limited to this example. For example, the argument map IM of time n ｎ This may include a value (material ID) for each pixel indicating the material type of each object O. That is, the argument map IM at time n ｎ This may be a material ID map. In this case, the given function is the argument map IM at time n. ｎ The function value map acquisition unit 225 may include a table T that associates the value of each pixel contained in with the function value (Figure 8). The function value map acquisition unit 225 then refers to the table T and obtains the argument map IM for time n. ｎ You may obtain a function value from the value of each pixel contained in the array, and obtain a function value map at time n that has a function value for each pixel and the number of input pixels.
[0071] Furthermore, in this embodiment and the above modified example, the argument map IM of time n ｎ However, the example given relates to the surface characteristics of each object O, but it is not limited to this. Furthermore, in this embodiment and the above modified example, the argument map IM at time n ｎ However, the drawing basic information map RM of multiple types at time n ｎ The example given was one of the cases, but the argument map IM for time n ｎ This is a map RM containing multiple types of basic drawing information for time n. ｎ Two or more of these may be included. For example, the argument map acquisition unit 224 acquires the argument map IM for time n. ｎ A transparency map and a depth map may be obtained. In that case, the predetermined function may include, for example, an operation to determine whether the depth of each pixel in the depth map at time n is less than or equal to a predetermined threshold, and a conversion expression to convert the transparency of the pixel in the transparency map into a function value for pixels whose depth is determined to be less than or equal to the predetermined threshold.
[0072] Furthermore, although this embodiment illustrates a case where the image processing system 1 is composed of a game console, i.e., a single computer, the image processing system 1 may be composed of multiple computers. For example, the image processing system 1 may be configured by connecting a server and a game console via a network such as the Internet.
[0073] Furthermore, in this embodiment, the image processing system 1 is illustrated as including a rendering system 20, which is realized by the control unit 10 executing a rendering program, and an estimation system 22, which is realized by the control unit 10 executing an estimation program. However, the functions of the rendering system 20 and the estimation system 22 may be realized by a single program.
[0074] Furthermore, in this embodiment, the frame f to be processed at time n is f ｎ Based on this, input frame F at time n has a number of input pixels greater than the initial number of pixels. ｎ The frame f of time n is obtained, but the frame to be processed is f ｎThat itself is input frame F at time n. ｎ It may be used as such. In other words, the number of input pixels must be greater than or equal to the initial number of pixels (i.e., the number of input pixels must be the same as or greater than the initial number of pixels).
[0075] Furthermore, in this embodiment, the processing target frame generation unit 2024 generates the processing target frame f at time n. ｎ The rendering information is stored in the rendering information storage unit 24, but the frame to be processed at time n may be stored in an external storage medium or the like. In that case, the input frame acquisition unit 220 refers to the storage medium or the like and selects the frame to be processed at time n f ｎ You may obtain the following. Similarly, multiple types of drawing basic information maps RM for time m. ｍ The data may also be stored on an external storage medium. Similarly, the machine learning model M may also be stored on an external storage medium.
[0076] Furthermore, in this embodiment, the auxiliary information map acquisition unit 223 acquires the function value map OM for time n. ｎ Based on the function value of each pixel contained in the input frame F at time n, ｎ The value of each pixel and the cumulative feature information map FM at time n-1. ｎ-1 By alpha blending the values of each pixel, the auxiliary information map AM at time n-1 is obtained. ｎ-1 The value of each pixel is calculated, but the input frame F at time n ｎ Instead of the value of each pixel, a predetermined value (e.g., 0) is used in the cumulative feature information map FM at time n-1. ｎ-1 The values of each pixel may also be blended with alpha.
[0077] Alternatively, the auxiliary information map acquisition unit 223 determines whether the function value of each pixel included in the function value map at time n is below a predetermined threshold, and if it determines that the function value of the pixel is below the predetermined threshold, the cumulative feature information map FM at time n-1 ｎ-1 The value of the pixel at time n in the input frame F ｎ By substituting the value of the corresponding pixel, the auxiliary information map AM at time n-1 ｎ-1 The value of each pixel may also be calculated.
[0078] Furthermore, in this embodiment, the auxiliary information map acquisition unit 223 inputs time n to the input frame Fｎ Function value map OM of the value of each pixel and time n ｎ The product of the value of the pixel and the cumulative feature information map FM at time n-1 ｎ-1 The value of the corresponding pixel and the function value map OM from 1 to time n ｎ The product of the value obtained by subtracting the value of the pixel and the sum of the two values is used in the auxiliary information map AM at time n-1. ｎ-1 The value of the corresponding pixel is calculated, but the input frame F at time n ｎ The value of each pixel and the function value map OM from 1 to time n ｎ The product of the value obtained by subtracting the value of the pixel and the cumulative feature information map FM at time n-1 ｎ-1 The function value map OM of the value of the pixel and time n ｎ The product of the value of the corresponding pixel and the sum of the two values are used in the auxiliary information map AM at time n-1. ｎ-1 It may also be calculated as the value of the corresponding pixel.
[0079] Furthermore, in this embodiment, at least the cumulative feature information map FM at time n-1 relating to the features of the input frame at time n-1 ｎ－１ Based on time n-1, the auxiliary information map AM ｎ-1 It obtains the auxiliary information map AM at time n-1 based on the input frame itself at time n-1. ｎ-1 It is also possible to obtain the following: That is, one or more processors may perform a past information map acquisition step in which they acquire the input frame at time n-1 as a past information map at time n-1, and in the auxiliary information map acquisition step, they may obtain the auxiliary information map at time n-1 by applying a contribution adjustment mask for time n (a function value map for time n) to this past information map at time n-1. In this embodiment, the cumulative feature information map FM at time n-1 ｎ－１ This corresponds to the past information map at time n-1, and the processing performed by the estimation unit 221 corresponds to the past information map acquisition step.
[0080] [5. Addendum] (1) An image processing system comprising at least one memory for storing one or more programs, and at least one processor, wherein the at least one processor sequentially generates processing target frames for times 1 to N (where N is a natural number of 2 or more) having the initial number of pixels, based on a plurality of drawing basic information maps for times 1 to N (where N is a natural number of 2 or more), each having a value for determining the color of a plurality of drawing target pixels, each of which is arranged in a virtual space and each of which is represented by three-dimensional data, according to the one or more programs; an input frame acquisition step of acquiring an input frame for time n having an input pixel count equal to or greater than the initial number of pixels, based on the processing target frame for time n (where n is a natural number of 2 or more and less than or equal to N); and a past information map acquisition step of acquiring a past information map for time n-1 having the input pixel count, based on the input frame for time n-1 at least. The process includes: an auxiliary information map acquisition step to acquire an auxiliary information map for time n-1 which is input to the machine learning model together with the input frame for time n, by applying a contribution adjustment mask for time n that indicates how much the value of each pixel included in the past information map for time n-1 contributes to the estimation by the machine learning model; an estimation step to acquire an estimated frame for time n based on the input frame for time n, the auxiliary information map for time n-1, and the machine learning model; and the process of acquiring an argument map for time n as an argument map for time n according to one or more programs, and the process of acquiring a function value map for time n which has the function value for each pixel and the number of input pixels by applying a predetermined function to the value of each pixel included in the argument map for time n-1. Image processing system.(2) The image processing system according to (1), wherein the one or more programs include a rendering program and an estimation program, the at least one processor executes the rendering step according to the rendering program, and further executes a providing step at each time n, which provides the estimation program with at least one of the plurality of basic drawing information maps for time n and the frame to be processed for time n, and the at least one processor executes the input frame acquisition step, the auxiliary information map acquisition step, the estimation step, the argument map acquisition step, and the function value map acquisition step according to the estimation program. (3) The image processing system according to (1) or (2), wherein the rendering step includes: a rendering target pixel determination step of determining the plurality of rendering target pixels at time m (where n is a natural number between 1 and N); a rendering basic information map acquisition step of acquiring the plurality of rendering basic information maps at time m based on the three-dimensional data representing each of the objects; and a processing target frame generation step of generating the processing target frame at time m based on the plurality of rendering basic information maps at time m. (4) The image processing system according to any one of (1) to (3), wherein the argument map at time n relates to the surface characteristics of each of the objects. (5) The image processing system according to (4), wherein the argument map at time n has a value for each of the pixels indicating the transparency of each of the objects, and the predetermined function includes a conversion formula that converts the value of each of the pixels included in the argument map at time n into the function value. (6) The image processing system according to (4), wherein the argument map at time n has a value for each pixel indicating the type of material of each object, and the predetermined function includes a table that associates the values of each pixel included in the argument map at time n with the function value.(7) The image processing system according to any one of (1) to (6), wherein the at least one processor calculates the value of each pixel in the auxiliary information map at time n-1 in the auxiliary information map acquisition step, based on the function value of each pixel in the function value map at time n, using a weighted average of the value of each pixel in the input frame at time n and the value of each pixel in the past information map at time n-1.
Claims
1. An image processing system comprising at least one memory for storing one or more programs, and at least one processor, wherein the at least one processor sequentially generates processing target frames for times 1 to N (where N is a natural number of 2 or more) having the initial number of pixels, based on a plurality of drawing basic information maps for times 1 to N (where N is a natural number of 2 or more), each having a value for determining the color of a plurality of drawing target pixels, each of which is arranged in a virtual space and each of which is represented by a plurality of three-dimensional data, and each having a predetermined initial number of pixels; an input frame acquisition step for acquiring an input frame for time n having an input number of pixels greater than or equal to the initial number of pixels, based on the processing target frame for time n (where n is a natural number of 2 or more and N or less); and a past information map acquisition step for acquiring a past information map for time n-1 having the input number of pixels, based on the input frame for time n-1 at least; The process includes: an auxiliary information map acquisition step to acquire an auxiliary information map for time n-1 which is input to the machine learning model together with the input frame for time n, by applying a contribution adjustment mask for time n that indicates how much the value of each pixel included in the past information map for time n-1 contributes to the estimation by the machine learning model; an estimation step to acquire an estimated frame for time n based on the input frame for time n, the auxiliary information map for time n-1, and the machine learning model; and the process of acquiring an argument map for time n as an argument map for time n according to one or more programs, and the process of acquiring a function value map for time n which has the function value for each pixel and the number of input pixels by applying a predetermined function to the value of each pixel included in the argument map for time n-1. Image processing system.
2. The image processing system according to claim 1, wherein the one or more programs include a rendering program and an estimation program, the at least one processor performs the rendering step according to the rendering program, and further performs a providing step at each time n, providing the estimation program with at least one of the plurality of basic drawing information maps for time n and the frame to be processed for time n, and the at least one processor performs the input frame acquisition step, the auxiliary information map acquisition step, the estimation step, the argument map acquisition step, and the function value map acquisition step according to the estimation program.
3. The image processing system according to claim 1 or 2, wherein the rendering step includes: a rendering target pixel determination step of determining the plurality of rendering target pixels at time m (where n is a natural number between 1 and N); a rendering basic information map acquisition step of acquiring the plurality of rendering basic information maps at time m based on the three-dimensional data representing each of the objects; and a processing target frame generation step of generating the processing target frame at time m based on the plurality of rendering basic information maps at time m.
4. The image processing system according to claim 1 or 2, wherein the argument map for time n relates to the surface characteristics of each object.
5. The image processing system according to claim 4, wherein the argument map at time n has a value for each pixel indicating the transparency of each object, and the predetermined function includes a conversion formula that converts the value of each pixel included in the argument map at time n into the function value.
6. The image processing system according to claim 4, wherein the argument map at time n has a value for each pixel indicating the type of material of each object, and the predetermined function includes a table that associates the values of each pixel included in the argument map at time n with the function value.
7. The image processing system according to claim 1 or 2, wherein the at least one processor, in accordance with one or more programs, in the auxiliary information map acquisition step, calculates the value of each pixel in the auxiliary information map at time n-1 based on a weighted average of the value of each pixel in the input frame at time n and the value of each pixel in the past information map at time n-1, based on the function value of each pixel included in the function value map at time n.
8. An image processing system for executing processing on processing target frames of time 1 to N (where N is a natural number of 2 or more), which are sequentially generated based on multiple types of drawing basic information maps for time 1 to N (where N is a natural number of 2 or more), each of which has a value for determining the color of each of the multiple drawing target pixels, each having a predetermined initial number of pixels, and which has the initial number of pixels, wherein the image processing system includes at least one memory for storing one or more programs, and at least one processor, wherein the at least one processor performs, according to the one or more programs, an input frame acquisition step of acquiring an input frame of time n having an input number of pixels greater than or equal to the initial number of pixels, based on the processing target frame of time n (where n is a natural number of 2 or more and N or less), and a past information map acquisition step of acquiring a past information map of time n-1 having the input number of pixels, based on the input frame of time n-1 at least, The process includes: an auxiliary information map acquisition step to acquire an auxiliary information map for time n-1 which is input to the machine learning model together with the input frame for time n, by applying a contribution adjustment mask for time n that indicates how much the value of each pixel included in the past information map for time n-1 contributes to the estimation by the machine learning model; an estimation step to acquire an estimated frame for time n based on the input frame for time n, the auxiliary information map for time n-1, and the machine learning model; and the process of acquiring an argument map for time n as an argument map for time n according to one or more programs, and the process of acquiring a function value map for time n which has the function value for each pixel and the number of input pixels by applying a predetermined function to the value of each pixel included in the argument map for time n-1. Image processing system.
9. A rendering step of sequentially generating processing target frames for times 1 to N (where N is a natural number of 2 or more) having the initial number of pixels, based on multiple types of drawing basic information maps for times 1 to N (where N is a natural number of 2 or more), each of which has a value for determining the color of each of the multiple drawing target pixels, each having a predetermined initial number of pixels; an input frame acquisition step of acquiring an input frame for time n having an input number of pixels greater than or equal to the initial number of pixels, based on the processing target frame for time n (where n is a natural number of 2 or more and N or less); a past information map acquisition step of acquiring a past information map for time n-1 having the input number of pixels, based on the input frame for time n-1 at least; and an auxiliary information map acquisition step of acquiring an auxiliary information map for time n-1 that is input to the machine learning model together with the input frame for time n, by applying a contribution adjustment mask for time n that indicates how much the value of each pixel included in the past information map for time n-1 contributes to the estimation by the machine learning model, to the past information map for time n-1. An image processing method comprising: an estimation step of obtaining an estimated frame at time n based on the input frame at time n, the auxiliary information map at time n-1, and the machine learning model; an argument map acquisition step of obtaining at least one of the plurality of basic drawing information maps at time n as an argument map at time n; a function value map acquisition step of obtaining a function value by applying a predetermined function to the value of each pixel included in the argument map at time n, and obtaining a function value map at time n having the function value for each pixel and the number of input pixels; wherein in the auxiliary information map acquisition step, the function value map at time n is used as the contribution adjustment mask at time n-1.
10. A rendering step in which processing target frames for times 1 to N (where N is a natural number of 2 or more) having the initial number of pixels is sequentially generated based on multiple types of drawing basic information maps for times 1 to N (where N is a natural number of 2 or more), each of which has a value for determining the color of each of the multiple drawing target pixels, each having a predetermined initial number of pixels; an input frame acquisition step in which an input frame for time n having an input number of pixels greater than or equal to the initial number of pixels is obtained based on the processing target frame for time n (where n is a natural number of 2 or more and N or less); a past information map acquisition step in which a past information map for time n-1 having the input number of pixels is obtained based on the input frame for time n-1 at least; and an auxiliary information map acquisition step in which an auxiliary information map for time n-1 is obtained by applying a contribution adjustment mask for time n to the past information map for time n-1, which indicates how much the value of each pixel included in the past information map for time n-1 contributes to the estimation by the machine learning model, to the past information map for time n-1, thereby obtaining an auxiliary information map for time n-1 that is input to the machine learning model together with the input frame for time n. A program that causes a computer to perform an estimation step of obtaining an estimated frame at time n based on the input frame at time n, the auxiliary information map at time n-1, and the machine learning model; an argument map acquisition step of obtaining at least one of the plurality of basic drawing information maps at time n as an argument map at time n; a function value map acquisition step of obtaining a function value by applying a predetermined function to the value of each pixel included in the argument map at time n, and obtaining a function value map at time n having the function value for each pixel and the number of input pixels; and in the auxiliary information map acquisition step, the function value map at time n is used as the contribution adjustment mask at time n-1.