Information processing method, information processing device, and computer-readable non-transitory storage medium
Patent Information
- Application Number
- PCT/JP2026/005959
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-26
- Filing Date
- 2026-02-18
- Publication Date
- 2026-09-03
Smart Images
Figure JP2026005959_03092026_PF_FP_ABST
Abstract
Description
Information processing method, information processing device, and computer-readable non-temporary storage medium
[0001] This disclosure relates to an information processing method, an information processing apparatus, and a computer-readable non-temporary storage medium.
[0002] Conventionally, a super-resolution processing method called MFSR (Multi Frame Super Resolution) (hereinafter referred to as "image quality enhancement processing") has been known as a DNN (Deep Neural Network) processing method for video that has an RNN (Recurrent Neural Network) structure.
[0003] MFSR is a technique that performs super-resolution on the current frame by blending inference information about the current frame and past frames, while upscaling the resolution of the current frame as needed, and then performing super-resolution on the current frame based on the blending result. In such MFSR, a technique called "jittering" is used for the purpose of anti-aliasing in CG (Computer Graphics) rendering.
[0004] Jittering involves shifting the sampling position of pixels in each incoming current frame at the sub-pixel level. By integrating the sampling information of each pixel obtained through such jittering in the time direction, aliasing can be suppressed (see, for example, Patent Document 1).
[0005] Jittering is performed using a jitter pattern that represents the sampling position and its sequence for each time step corresponding to the current frame. While it's common to obtain the sampling position each time using random numbers, it's also common to use pseudo-random number sequences such as Halton sequences to generate the jitter pattern.
[0006] On the other hand, when jittering is performed, it is known that temporal flickering is likely to occur because the phase of the pixel sampling position shifts between frames. Therefore, a technique called "phase shift" is sometimes employed, in which the pixel value corresponding to the sampling value at the center position of each pixel is obtained by interpolation from the sampling values of surrounding pixels due to jittering, and the phase between frames is aligned at the center position.
[0007] Japanese Patent Publication No. 2023-067714
[0008] However, the conventional techniques described above still have room for further improvement in optimizing sharpness processing for the current frame.
[0009] For example, when the phase shift described above is performed, the pixel value of each pixel in the current frame is obtained by interpolation from surrounding pixels, which can cause blurring and a decrease in the perceived resolution of the current frame.
[0010] In this regard, it is conceivable to apply a sharpening process to the current frame after phase shifting to improve the perceived resolution. However, since the amount of pixel interpolation due to phase shifting differs from frame to frame due to jittering, the degree of blurring differs in each frame of the video. Therefore, simple sharpening processing may not provide sufficient sharpness throughout the entire video.
[0011] Therefore, this disclosure proposes an information processing method, an information processing device, and a computer-readable non-temporary storage medium that can optimize sharpness processing for the current frame.
[0012] To solve the above problems, one form of information processing method according to the present disclosure is an information processing method performed by a computer, which in DNN processing for video having an RNN structure, renders while shifting the phase of the sampling position of pixels for each current frame of a series of video frames, performs a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position, determines the sharpness intensity of sharpness processing for the current frame based on the difference in contrast between a history frame on which motion compensation has been performed on the DNN output result one frame before the current frame and the current frame after the phase shift, and performs the sharpness processing based on the sharpness intensity.
[0013] This is an explanatory diagram of a basic configuration example of DNN processing for video having an RNN structure. This is an explanatory diagram of jittering (part 1). This is an explanatory diagram of jittering (part 2). This is an explanatory diagram of phase shift. This is an explanatory diagram of a configuration example when sharpness processing is performed. This is an explanatory diagram of the difference in sampling position due to jittering. This is an explanatory diagram of an image processing method according to an embodiment of this disclosure. This is a block diagram showing an example of the configuration of an image processing apparatus according to an embodiment of this disclosure. This is an explanatory diagram of sharpness processing. This is an explanatory diagram of sharpness intensity analysis processing (part 1). This is an explanatory diagram of sharpness intensity analysis processing (part 2). This is an explanatory diagram of sharpness intensity analysis processing (part 3). This is a detailed block diagram of an image processing apparatus according to an embodiment of this disclosure. This is a flowchart showing the processing procedure of the processing performed by the image processing apparatus. This is a flowchart showing the processing procedure of sharpness intensity analysis processing. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the image processing apparatus.
[0014] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.
[0015] Furthermore, in the following, the information processing device according to the embodiment of this disclosure (hereinafter referred to as "this embodiment") shall be an image processing device 10. The information processing method according to this embodiment shall be an image processing method executed by the control unit 12 (see Figure 8) of this image processing device 10.
[0016] Furthermore, in the following, the video frame that is to be processed among the video frames input to the image processing device 10 will be referred to as the "current frame" or "current image" as appropriate. Also, the image obtained by applying motion compensation to the DNN output of the frame before the current frame will be referred to as the "history frame" or "history image" as appropriate. Also, "motion compensation" will be referred to as "MC" as appropriate. "MC" is an abbreviation for "Motion Compensation".
[0017] Furthermore, this disclosure will be explained in the following order of items: 1. Overview 1-1. Basic Configuration Example 1-2. Issues of Existing Technologies 1-3. Overview of the Image Processing Method According to the Embodiment of this Disclosure 2. Configuration Example of the Image Processing Device 2-1. Explanation of Sharpness Processing 2-2. Explanation of Sharpness Intensity Analysis Processing 2-3. Detailed Explanation of the Block Diagram 3. Processing Procedure 4. Modifications 4-1. First Modification 4-2. Second Modification 4-3. Other Modifications 5. Hardware Configuration 6. Conclusion
[0018] <<1. Overview>> <1-1. Basic Configuration Example> First, let's describe the background for explaining this embodiment in more detail. Figure 1 is an explanatory diagram of a basic configuration example of DNN processing for video having an RNN structure.
[0019] For video DNN processing, the RNN structure is common. As shown in Figure 1, the RNN structure is a DNN processing structure that simultaneously inputs the current image obtained from the renderer and the post-MC history image from the MC unit (an image obtained by applying MC processing to the DNN output of the previous frame).
[0020] This DNN structure processes using two consecutive frames, making it easier to output video in a way that minimizes the difference in pixel values between the current image and the history image. ｔ This makes it easier to maintain consistency in rendered images. As a result, rendered images output via DNN processing for video with an RNN structure are generally known to have high temporal stability.
[0021] Specifically, rendered images are prone to temporal flickering due to factors such as jittering and ray tracing noise. However, by using an RNN structure, it becomes possible to internally analyze the differences in pixels between frames, thereby reducing this flickering.
[0022] The DNN undergoes pre-training, and in the case of super-resolution, it uses low-resolution video (student data) and high-resolution video (training data) to pre-generate learning coefficients (neural network weights) so that when low-resolution video is input, it can output higher-resolution video. As shown in Figure 1, in the following, the DNN according to this embodiment will be assumed to operate in the image quality processing unit.
[0023] The renderer performs rendering that includes jittering. Figure 2 is an explanatory diagram (part 1) of jittering. In Figure 2, the horizontal axis "JitterX" indicates the horizontal position of the pixel PX. The vertical axis "JitterY" indicates the vertical position of the pixel PX. The horizontal and vertical positions are shown normalized by setting the horizontal and vertical sizes of the pixel PX to 1, respectively.
[0024] In rendering that includes jittering, the sampling position is set randomly in the horizontal and vertical directions. Jittering shifts the sampling position of pixels (PX) over time at the subpixel level, and the sampling information is added together over time.
[0025] A clearer example is shown in Figure 3. Figure 3 is an explanatory diagram of jittering (part 2). As shown in Figure 3, a virtual CG camera is assumed to capture images of the CG space in which CG objects exist. In other words, the CG camera in this case is a CG renderer that samples each pixel value on the rendering plane corresponding to one frame of the image and generates a rendered image.
[0026] At this time, the CG camera acquires pixel values by jittering, shifting the sampling position of the pixel values for each frame from time t-3 to t. In other words, when the CG camera acquires color in CG space, it acquires pixel values by shifting them from the center position of each pixel. This allows it to acquire pixel value information at the subpixel level.
[0027] Based on the acquired pixel values, rendering images with different phases are generated for each frame from time t-3 to t. By temporally combining these rendering images using a DNN, a high-definition rendering image with suppressed aliasing can be obtained.
[0028] On the other hand, as is clear from the rendering image shown in Figure 3, the phase shifts between frames, making temporal flickering likely to occur. In DNNs that receive such images as input, they are sometimes trained beforehand using similar input data to suppress temporal flickering caused by jittering, but in reality, it is difficult to completely eliminate flickering. For example, even if a DNN tries to suppress the phase shift by performing temporal frame synthesis, considering that the subject and camera move, there are countless combinations of synthesis, making it difficult.
[0029] Therefore, phase shifting can be considered. Figure 4 is an explanatory diagram of phase shifting. As shown in "Calculation of Center Position" in Figure 4, in phase shifting, the pixel value corresponding to sampling at the center position is calculated by interpolation from the pixel values of surrounding pixels using jittering. Note that Figure 4 shows an example of bilinear interpolation using the four surrounding pixels, but bicubic interpolation using the sixteen surrounding pixels may also be used.
[0030] In other words, phase shifting aligns the phase of each frame of the rendered image at the center position of each pixel by interpolating the pixel value corresponding to the sampling at the center position from surrounding pixels. As a result, each frame of the rendered image, with its phase aligned at the center position of the pixels, is input to the DNN, making temporal flickering less likely to occur.
[0031] On the other hand, since phase shifting is performed by pixel interpolation, it can cause blurring in the rendered image, and the perceived resolution of the high-resolution processing results output by the DNN tends to decrease. Furthermore, in phase shifting, interpolation is weighted according to the distance from the center position of the pixel, so the blurring tends to become stronger as the sampling position due to jittering moves further away from the center position.
[0032] Therefore, sharpening processing can be considered. Figure 5 is an explanatory diagram of an example configuration when sharpening processing is performed. When sharpening processing is performed, as shown in Figure 5, the sharpening processing unit applies sharpening processing to the current image after the phase shift processing unit has performed the phase shift processing. The sharpening processing unit then outputs the current image after sharpening to the image quality enhancement processing unit.
[0033] As a result, the DNN in the high-image-quality processing unit takes a high-definition current image with the blur caused by phase shift suppressed as input, and outputs an image that is the result of high-image-quality processing by the DNN. ｔ The perceived resolution will improve.
[0034] <1-2. Challenges of Existing Technologies> However, even when performing sharpness processing, if the sharpness intensity, which indicates the strength of sharpness, is too high, there is a problem that the image may be over-emphasized or distorted. Also, if the sharpness intensity is too low, blurring is not sufficiently suppressed, and it is difficult to obtain the effect of improving resolution.
[0035] In addition, when using phase shift to suppress flickering caused by jitter, as mentioned above, the phase shift interpolates weighted according to the distance from the center position of the pixel, so the degree of blur tends to vary from frame to frame. For example, a frame at time t-1 may have almost no blur, but the frame at the next time t may suddenly have a strong blur.
[0036] Figure 6 is an explanatory diagram of the difference in sampling position due to jittering. As shown in Figure 6, the sampling position SP due to jittering １ The sampling position SP ２ Compared to that, the pixel PX is assumed to be relatively farther from the center position CP.
[0037] In this case, sampling position SP １ For this case, the amount of shift in the phase shift becomes large, and pixel interpolation is performed in a way that strongly averages the surrounding pixels, making it prone to blurring. In this case, unless the sharpness intensity is increased, it is difficult to obtain the effect of improving the perceived resolution through sharpness processing.
[0038] On the other hand, sampling position SP ２ For this case, the amount of shift in the phase shift is small, and pixel interpolation is performed in a way that does not involve much averaging of surrounding pixels (i.e., the pixel value of the nearest sampling position is heavily weighted), making it less prone to blurring. In this case, if the sharpness intensity is set too high, the image may become over-emphasized, with excessive edge enhancement or even distortion.
[0039] It should be noted that similar issues can occur not only with jittering, but also with renderers that have a Dynamic Resolution (DR) function. DR is a function that dynamically controls the rendering resolution according to the computational load during game execution, for example. When the load is high, it renders the image at a relatively lower resolution to avoid image delays and freezes that can occur under heavy load.
[0040] When DR is enabled, depending on the game's load, it's possible that low-resolution rendering may continue for many frames, only for a high-resolution rendering image to suddenly become input. In this situation, if a strong sharpness setting is applied to match the blurry low-resolution image, there is a risk of over-sharpening the high-resolution image.
[0041] <1-3. Overview of the Image Processing Method According to the Embodiment of the Present Disclosure> In the image processing method according to the embodiment of the present disclosure, the control unit 12 of the image processing apparatus 10 performs rendering in DNN processing for video having an RNN structure, shifting the phase of the pixel sampling position for each current frame of a series of video frames, performing a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position, determining the sharpness intensity for sharpness processing on the current frame based on the difference in contrast between the history frame, which has motion compensation applied to the DNN output result one frame before the current frame, and the current frame after the phase shift, and performing sharpness processing based on the sharpness intensity.
[0042] Figure 7 is a schematic diagram illustrating the image processing method according to this embodiment. As shown in Figure 7, the image processing method according to this embodiment includes a sharpness intensity analysis unit that performs sharpness intensity analysis, in addition to the configuration example for sharpness processing shown in Figure 5.
[0043] The sharpness intensity analysis unit calculates the optimal sharpness intensity for each pixel of the current image based on the differences in characteristics between the current image after phase shifting and the history image after multiplication correction (MC). The sharpness intensity analysis unit then outputs the calculated sharpness intensity to the sharpness processing unit. The sharpness processing unit then applies sharpening processing to the current image after phase shifting where blurring occurs, based on the optimal sharpness intensity obtained from the sharpness intensity analysis unit.
[0044] The sharpness processing unit then outputs the current image after sharpening, with blur suppressed by sharpness processing based on the optimal sharpness intensity, to the image quality enhancement processing unit. In the image quality enhancement processing unit, the DNN takes the current image with blur suppressed by the optimal sharpness intensity as input, and the output image is the result of high-quality processing by the DNN. ｔ The perceived resolution will improve.
[0045] Details of each process performed by the image processing device 10, including the sharpness intensity analysis process, will be described later with reference to Figure 8 and subsequent figures.
[0046] As described above, in the image processing method according to this embodiment, the control unit 12 of the image processing apparatus 10 performs rendering in DNN processing for video having an RNN structure, shifting the phase of the pixel sampling position for each current frame of a series of video frames, performing a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position, determining the sharpness intensity for sharpening processing on the current frame based on the difference in contrast between the history frame, which has motion compensation applied to the DNN output result one frame before the current frame, and the current frame after the phase shift, and performing sharpening processing based on the sharpness intensity.
[0047] Therefore, the image processing method according to this embodiment makes it possible to optimize sharpness processing for the current frame. Below, an example of the configuration of an image processing device 10 to which the image processing method according to this embodiment is applied will be described in more detail.
[0048] <<2. Example of Image Processing Device Configuration>> Figure 8 is a block diagram showing an example of the configuration of the image processing device 10 according to this embodiment. As shown in Figure 8, the image processing device 10 comprises a storage unit 11 and a control unit 12. The image processing device 10 is also connected to an input unit 3 and an output unit 5.
[0049] The input unit 3 is a component that receives input to the image processing device 10. The input unit 3 receives input such as the current image to be processed, motion vectors, and camera parameters.
[0050] The current image refers to a single frame within a sequence of consecutive frames, such as in a video. The current image is a two-dimensional RGB image acquired from a camera. The camera can be a virtual camera used as a CG camera, or a real camera used in virtual production (VP), etc.
[0051] A motion vector is a two-dimensional vector that defines the movement of corresponding pixels between frames, defined by the amount of pixel movement (number of pixels) in the horizontal and vertical directions. Motion vectors can be obtained through a general estimation process using a series of consecutive frames.
[0052] Camera parameters include information such as the camera's position, orientation, field of view, and lens parameters in the virtual space. Based on the camera's position and orientation information, camera movement (direction of movement, amount of movement, etc.) can be detected.
[0053] Furthermore, the input unit 3 accepts input of various settings related to rendering (such as jitter pattern, resolution, ray tracing settings, and other general settings). The input unit 3 also accepts input from the user, such as instructing the execution of rendering. When the input unit 3 accepts input from the user, it may be implemented using a keyboard, pointing device, or game controller. The input unit 3 also outputs the various types of information it has received to the image processing device 10.
[0054] The output unit 5 is a component that displays visual information such as output images output from the image processing device 10. The output unit 5 is implemented by a display, an image projection device, or the like.
[0055] The memory unit 11 is implemented by a memory device such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, or HDD (Hard Disk Drive).
[0056] In the example shown in Figure 8, the memory unit 11 stores DNN model information 11a and history information 11b. The DNN model information 11a is information including the DNN model used in the image quality processing unit 12f, which will be described later. This DNN model is a deep learning model that learns and infers how to improve the image quality of the current image. It is desirable that this DNN model be implemented as a deep learning model that is trained to minimize the loss of both the image quality processing result, which is the output when the current image and history image are input, and the GT (Ground Truth) image that is paired with the image quality processing result, using the steepest descent method or the like.
[0057] History information 11b stores various information about the history image. For example, history information 11b stores history information about history frames for a predetermined period from the current frame being processed to the most recent past.
[0058] The control unit 12 controls each part of the image processing device 10. The control unit 12 is implemented, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphical Processing Unit), APU (Accelerated Processing Unit), etc., which executes a program according to this embodiment (not shown) stored in the storage unit 11 using RAM as the working area. Alternatively, the control unit 12 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).
[0059] The control unit (12) includes an acquisition unit (12a), a renderer (12b), a scaler (12c), a phase shift processing unit (12d), a sharpness processing unit (12e), an image quality improvement processing unit (12f), an MC unit (12g), a sharpness intensity analysis unit (12h), and an output control unit (12i).
[0060] Among the respective units of the control unit (12), the description with reference to FIG. 8 will detail the sharpness processing unit (12e) and the sharpness intensity analysis unit (12h). Details of the relationship between blocks including units other than the sharpness processing unit (12e) and the sharpness intensity analysis unit (12h) will be described later with reference to the detailed block diagram of FIG. 13.
[0061] The acquisition unit (12a) acquires an input from the input unit 3. The acquisition unit (12a) also outputs the input from the input unit 3 to the renderer (12b) as an input signal. The renderer (12b) renders the current image and the motion vector with a phase shifted by jittering.
[0062] The scaler (12c) enlarges the resolution of the current image and the motion vector (which can be read as the number of pixels) as necessary. The phase shift processing unit (12d) performs the above-described phase shift processing on the current image and the motion vector.
[0063] <2-1. Description of Sharpness Processing> The sharpness processing unit (12e) executes sharpness processing with adjustable sharpness intensity. FIG. 9 is an explanatory diagram of sharpness processing. As shown in FIG. 9, for example, consider a case where a pixel value of a target pixel i is determined using a 3×3 pixel filter F.
[0064] Here, the pixel value i of the target pixel i ４ , for the pixel value, the pixel values of N surrounding pixels (N is a natural number of 1 or more) are i ０ , ..., i Ｎ－１ , and let w be the value of the sharpness intensity. In the example of FIG. 9, since the filter F has 3×3 pixels, N=9. In this case, if the pixel value of the pixel position of the target pixel i after sharpness processing is i', the pixel value i' is expressed by the following formula (a).
[0065]
[0066] According to equation (a), the degree of enhancement in sharpness processing increases in proportion to the value of sharpness intensity w. The sharpness intensity analysis unit 12h calculates the respective sharpness intensity w for each pixel of each current image in the sharpness intensity analysis process described later. Here, an example is given where the filter F is 3x3 pixels, but the filter size is not limited to this.
[0067] Returning to the explanation of Figure 8, the image quality processing unit 12f reads the DNN model contained in the DNN model information 11a and operates it as DNN12fa. The image quality processing unit 12f also inputs the current image after sharpening processing and the history image to DNN12fa and obtains the image quality processing result from the DNN. DNN12fa takes the current image and the history image as input, infers the image quality processing result of the current image, and outputs it. The MC unit 12g performs MC processing based on the history image and motion vector output from DNN12fa as the image quality processing result.
[0068] <2-2. Explanation of Sharpness Intensity Analysis Process> The sharpness intensity analysis unit 12h performs sharpness intensity analysis. Figure 10 is an explanatory diagram (1) of the sharpness intensity analysis process. Figure 11 is an explanatory diagram (2) of the sharpness intensity analysis process. Figure 12 is an explanatory diagram (3) of the sharpness intensity analysis process.
[0069] In the sharpness intensity analysis process, a sharpness intensity w is calculated for each pixel of the current image. This allows for sharpening processing with different optimal sharpness intensities for each pixel.
[0070] As shown in Figure 10, in the sharpness intensity analysis process, the pixel i of interest in the current frame from which the sharpness intensity w is to be calculated is considered. Ｃ Regarding this, the corresponding pixel i in the history frame Ｈ The pixel values, and the local region R of both frames. Ｃ , R Ｈ The sharpness intensity w is calculated using the pixel values.
[0071] The sharpness intensity analysis unit 12h first analyzes the local region R of both frames. Ｃ , R Ｈ The sharpness intensity w is determined based on the difference in contrast. Next, the sharpness intensity analysis unit 12h analyzes the pixel i of interest in both frames. Ｃ i Ｈ The difference in brightness values, and the local region R of the current frame. Ｃ The value of sharpness intensity w is modified (for example, suppressed to 0) based on the variance of the luminance values.
[0072] Let me explain in detail. First, the sharpness intensity analysis unit 12h calculates the sharpness intensity w according to the local contrast difference between the current frame and the history frame. At this time, the sharpness intensity analysis unit 12h determines the sharpness intensity w of the current frame based on the contrast of the history frame, which is a high-quality image after the quality enhancement processing. In other words, the sharpness intensity analysis unit 12h determines the sharpness intensity so that the contrast difference between the history frame and the current frame is small. This makes it possible to avoid setting an excessive or insufficient sharpness intensity w.
[0073] More specifically, as shown in Figure 11, the sharpness intensity analysis unit 12h analyzes the pixel of interest i of the current frame and the history frame. Ｃ i Ｈ Surrounding local region R Ｃ , R Ｈ Calculate the contrast for each region. Local region R Ｃ , R Ｈ For example, the filter size can be set to 3x3 pixels, similar to the filter size mentioned earlier.
[0074] Here, in the local region R Ｃ The contrast of c Ｃ , the contrast of the local RH region c Ｈ Let's assume that contrast c Ｃ , c Ｈ For example, the brightness value i of the pixels in both frames. Ｙ ,j Ｙ It can be determined from the ratio of the minimum and maximum values, and can be expressed by the following equation (b).
[0075]
[0076] Furthermore, the conversion from RGB to brightness values can be done using a general conversion formula, and the RGB values of a given pixel can be converted to i Ｒ i Ｇ i Ｂ Therefore, it can be expressed by the following equation (c).
[0077]
[0078] Sharpness intensity w is basically the contrast ratio c Ｃ / c Ｈ This can be determined by the formula, but the gain α and constant b can also be set as hyperparameters, in which case the formula becomes as shown in equation (d) below.
[0079]
[0080] The gain α and constant b may be determined by the system developers, etc., after evaluating the final output image quality of the DNN12fa in advance, so as to maximize image quality. Note that the function for determining sharpness intensity is not limited to the form of equation (d) above; for example, a quadratic function may also be used.
[0081] Next, the sharpness intensity analysis unit 12h modifies the sharpness intensity w according to the local difference in brightness. The sharpness intensity analysis unit 12h suppresses sharpness processing, for example, by changing the calculated sharpness intensity w to 0.
[0082] For example, so-called ray-tracing noise may occur in rendered images. In that case, if sharpening is applied to pixels containing noise, the noise will be amplified, and the output of DNN12fa may contain noise. Here, we give an example of avoiding noise amplified by estimating the presence or absence of noise for each pixel and setting the sharpness intensity w to 0 if noise is present.
[0083] To explain the specific procedure, as shown in Figure 12, the pixels where noise occurs in the current frame tend to have significantly different brightness values from the pixels at the same position in the history frame. Also, if there are noisy pixels, the local region R of the current frame... Ｃ The variance of the luminance values tends to be large. Therefore, the sharpness intensity analysis unit 12h uses the difference value d of the luminance values between the two frames and the local region R of the current frame. Ｃ The occurrence of noise is estimated from the variance value V of the luminance values.
[0084] Current frame's featured pixel i Ｃ and the same-positioned pixel i in the history frame Ｈ For each pixel value, i ４ ,j ４ , the brightness value i ４ Ｙ ,j ４ Ｙ Let's assume that the local region R of the current frame is also considered. Ｃ i ０ , ..., i Ｎ－１ Let's assume that in the example in Figure 12, N = 9. In this case, the difference d in luminance values between the current frame and the history frame can be expressed by the following equation (e).
[0085]
[0086] Furthermore, the local region R of the current frame Ｃ The variance V can be expressed by the following formula (f).
[0087]
[0088] Also, the notable pixel i Ｃ When the region is a noise area, the estimation is: difference value d > difference value threshold th ｄ , variance value V > variance value threshold th Ｖ This is done by evaluating whether the two criteria are satisfied simultaneously. Here, the difference value threshold th ｄ , variance value threshold th Ｖ is a hyperparameter, a threshold set in advance by the system designer or developer. When both the difference value d and the variance value V exceed the threshold, the pixel of interest i ＣThis is presumed to be noise. If it is presumed to be noise, the sharpness intensity w = 0, as shown in equation (g) below. If it is not presumed to be noise, the value of the sharpness intensity w does not change due to this process, as shown in equation (g) below, and the sharpness intensity w corresponding to the contrast, as described above, is used as is.
[0089]
[0090] Returning to the explanation of Figure 8, the output control unit 12i acquires the image quality processing result output from the image quality processing unit 12f as an output image. The output control unit 12i also performs output control, applying general post-processing such as color conversion and codec to the acquired output image and outputting it to the output unit 5.
[0091] <2-3. Detailed Explanation of the Block Diagram> Next, we will explain the details of the block diagram, including the data flow based on the explanations above. Figure 13 is a detailed block diagram of the image processing apparatus 10 according to this embodiment. Note that, for the sake of explanation, the acquisition unit 12a and the output control unit 12i are omitted from the illustration in Figure 13.
[0092] First, the renderer 12b receives the input signal S from the input unit 3. The input signal S, for example, controls the operation of the renderer 12b, instructing it to perform various rendering settings and execute rendering, as described above.
[0093] Furthermore, the renderer 12b renders the image according to the command of the input signal S, and the current image I Ｃ Then, the motion vector MV is obtained and sent to the scaler 12c. Also, the current image I Ｃ Furthermore, the rendering of motion vectors (MV) includes jitter, and pixel values are obtained from different phases for each frame.
[0094] Note that normally, current image I Ｃ The motion vector MV is output continuously over time. Also, jitter pattern information S is used during rendering. ＪThis is a numerical value indicating how much each pixel is shifted horizontally and vertically from its center position CP relative to the rendered image plane, at a sub-pixel level between -0.5 and +0.5. This is the jitter pattern information S. Ｊ This is sent to the phase shift processing unit 12d.
[0095] Scaler 12c will, if necessary, current image I Ｃ And the resolution (number of pixels) of the motion vector MV is enlarged, and the enlarged current image I ＣＲ and the motion vector MV after scaling Ｒ This is sent to the phase shift processing unit 12d.
[0096] Note that the number of pixels accepted by the DNN12fa architecture is fixed, and the current image I Ｃ Furthermore, if the number of pixels in the motion vector MV changes according to the rendering settings of renderer 12b, the number of pixels may be enlarged (upscaled) by a desired magnification so that the number of pixels matches the specifications of DNN12fa.
[0097] The upscaling method used at this time can be arbitrarily selected, for example, bilinear interpolation or bicubic interpolation. Regarding the motion vector (MV), the nearest neighbor method is preferable to avoid the generation of intermediate values due to pixel interpolation, but it is not particularly limited to this method.
[0098] The phase shift processing unit 12d processes jitter pattern information S Ｊ Current image I drawn based on ＣＲ and motion vector MV Ｒ In response, a phase shift process is performed to correct the deviation caused by jittering.
[0099] In the phase shift process, pixel values equivalent to those obtained when drawing at the center position CP without jittering are interpolated. Pixel interpolation methods such as bilinear interpolation or bicubic interpolation may be used. Regarding the motion vector MV, conversion to intermediate interpolated values is undesirable, so the nearest neighbor method is preferable.
[0100] The phase shift processing unit 12d receives jitter pattern information S from the renderer 12b.Ｊ By referring to this, the appropriate phase shift processing is applied to the frame to be processed. The phase shift processing unit 12d also processes the current image I after phase shifting. ＣＳ The data is sent to the sharpness processing unit 12e and the sharpness intensity analysis unit 12h. The phase shift processing unit 12d also processes the motion vector MV after the phase shift. Ｓ Send to MC section 12g.
[0101] The sharpness processing unit 12e processes the sharpness intensity S obtained from the sharpness intensity analysis unit 12h. Ｗ Based on this, current image I after phase shift ＣＳ The image is then sharpened, and the current image after sharpening is shown. Ｃ Get the current image I. Ｃ ' is sent to DNN12fa.
[0102] DNN12fa is the current image I after sharpening. Ｃ 'and history image after MC I Ｈ ＭＣ Input, output image I Ｏ This is sent to the output unit 5. Here, the history image I after MC is sent. Ｈ ＭＣ This is the current image I after sharpening. Ｃ ' corresponds to the image quality enhancement result from the previous frame by DNN12fa. Also, DNN12fa is the current image I after sharpening in the next frame. Ｃ For ', the output image I from the previous frame Ｏ This corresponds to historical image I. Ｈ Send to MC section 12g.
[0103] The coefficient values of DNN12fa are the pre-trained learning coefficient values described above. While the learning task for DNN12fa is primarily envisioned to be image quality enhancement such as super-resolution, this disclosure is not limited to this. This embodiment can be applied to general video DNN processing with an RNN structure, including tasks such as noise reduction and style transfer.
[0104] The MC unit 12g receives the motion vector MV from the phase shift processing unit 12d. ＳBased on the value of , perform general MC processing (motion compensation processing) on the history image I Ｈ . The motion vector MV Ｓ has a vector related to the spatial movement amount from the history image I Ｈ to the current image I ＣＳ , and the pixel positions of the history image I Ｈ and the current image I ＣＳ are matched by the MC processing. Further, the MC unit 12g sends the post-MC history image I Ｈ ＭＣ to the DNN 12fa and the sharpness intensity analysis unit 12h.
[0105] The sharpness intensity analysis unit 12h calculates an optimal sharpness intensity S ＣＳ from the post-phase-shift current image I Ｈ ＭＣ and the post-MC history image I Ｗ , and sends the optimal sharpness intensity S to the sharpness processing unit 12e. The output unit 5 acquires and outputs an output image I Ｏ that has been subjected to general post-processing such as color conversion and codec by an unillustrated output control unit 12i.
[0106] <<3. Processing Procedure>> Next, the processing procedure of processing executed by the image processing apparatus 10 will be described with reference to FIGS. 14 and 15. FIG. 14 is a flowchart showing the processing procedure of processing executed by the image processing apparatus 10. Further, FIG. 15 is a flowchart showing the processing procedure of sharpness intensity analysis processing.
[0107] As shown in FIG. 14, when the acquisition unit 12a acquires the input signal S from the input unit 3 (step S101), the renderer 12b performs rendering by shifting the phase of the current image I Ｃ and the motion vector MV using jitter (step S102).
[0108] Further, the control unit 12 applies the scaler 12c to the current image I Ｃ and the motion vector MV as necessary (step S103). Then, the phase shift processing unit 12d processes the rendered current image I ＣＲ to which the scaler 12c has been applied as necessary and the motion vector MVＲ Then, a phase shift process corresponding to the jitter is applied (step S104).
[0109] On the other hand, the MC section 12g has a motion vector MV after phase shift. Ｓ Based on this, historical image I Ｈ Movement vector MV Ｓ Applying this, the current image I after phase shift ＣＳ The phase is aligned (step S105).
[0110] Then, the sharpness intensity analysis unit 12h performs sharpness intensity analysis processing (step S106). As shown in Figure 15, in the sharpness intensity analysis processing, the sharpness intensity analysis unit 12h analyzes the current image I after phase shift. ＣＳ And history image I after the MC Ｈ ＭＣ Furthermore, local region R Ｃ , R Ｈ contrast c Ｃ , c Ｈ Calculate each of these (step S201).
[0111] Then, the sharpness intensity analysis unit 12h calculates the local region R Ｃ , R Ｈ contrast c Ｃ , c Ｈ Based on this, the sharpness intensity w is calculated (step S202). The sharpness intensity analysis unit 12h also analyzes the current image I ＣＳ The notable pixel i Ｃ And, history image I after the MC Ｈ ＭＣ The difference value d in brightness is calculated between the same position pixel and the other pixel (step S203).
[0112] Then, the sharpness intensity analysis unit 12h analyzes the current image I ＣＳ The notable pixel i Ｃ Local region R centered on Ｃ The variance value V of the luminance value is calculated (step S204). Then, the sharpness intensity analysis unit 12h determines that both the difference value d and the variance value V are at the threshold th ｄ ,th Ｖ It is determined whether or not it exceeds (step S205).
[0113] If it exceeds (Step S205, Yes), the sharpness intensity analysis unit 12h changes the sharpness intensity w to 0 (Step S206). If it does not exceed (Step S205, No), the sharpness intensity w remains unchanged. Then, the sharpness intensity analysis unit 12h sets the sharpness intensity w to the optimal sharpness intensity S Ｗ The output is then sent to the sharpness processing unit 12e (step S207), and the process is terminated.
[0114] Returning to the explanation of Figure 14, after the sharpness intensity analysis process, the sharpness processing unit 12e determines the optimal sharpness intensity S estimated in the sharpness intensity analysis process. Ｗ Sharpening processing is performed on the current image I ＣＳ Apply to (step S107). Then, the high-image-quality processing unit 12f processes the I after sharpness processing. Ｃ 'and history image I Ｈ ＭＣ This is input to DNN12fa (step S108).
[0115] Then, the output unit 5 outputs the image quality processing result from the image quality processing unit 12f as the output image I Ｏ In addition to acquiring it as (step S109), the control unit 12 outputs image I Ｏ History Image I Ｈ The RNN recursive processing will be repeated from step S105.
[0116] <<4. Modifications>> Although embodiments of the present disclosure have been described so far, several other modifications of the image processing method according to this embodiment can be given.
[0117] <4-1. First Modification> For example, regarding the determination of sharpness intensity w due to the difference in contrast in the sharpness intensity analysis unit 12h, the history image I Ｈ ＭＣ Alternatively, it may be determined in advance by comparing it with the GT image.
[0118] In this case, the training data used for pre-training DNN12fa is the GT image, and the current image I ＣＳLocal region R Ｃ contrast c Ｃ And, the contrast c of the local region in the GT image ＧＴ Therefore, we define a function that determines the sharpness intensity w in advance.
[0119] And in this case, for example, current image I after sharpening. Ｃ The contrast of the local region of ' Ｃ If we assume that, then c ＧＴ ≒c Ｃ Ideally, this would be the case. This is because if an image similar to the target GT image for the output of DNN12fa is input, the inference accuracy of DNN12fa will improve.
[0120] The formula for calculating the sharpness intensity w is "w = α(1 / c) Ｃ When using ") + b", the gain α and constant b are defined as the current image I after sharpening due to the sharpness intensity w. ＣＳ We need to find the gain α and constant b when the quantitative evaluation value (for example, PSNR (Peak Signal to Noise Ratio)) with respect to the GT image is maximized.
[0121] Furthermore, the gain α and constant b may be designed by exhaustive experimentation using a greedy algorithm, or statistically calculated using regression methods such as Bayesian optimization with the DNN12fa training dataset. In this case, the gain α and constant b are values calculated in advance before the system is put into actual operation, and during actual operation, the history image I Ｈ ＭＣ Do not refer to this. Note that for noise estimation, use current image I. ＣＳ and historical image I Ｈ ＭＣ Use the following: History image I Ｈ ＭＣ While it is possible to obtain better values by sequentially referencing the sharpness intensity w using this method, this approach has the advantage of reducing computational cost.
[0122] <4-2. Second Modification> Also, the current image I that is first input after the start of operation of the image processing device 10. ＣＳ This is historical image I Ｈ ＭＣSince it does not exist, sharpening processing can be omitted by setting the sharpness intensity w to 0.
[0123] <4-3. Other Modifications> Furthermore, among the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.
[0124] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.
[0125] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.
[0126] <<5. Hardware Configuration>> Furthermore, the image processing device 10 according to the above embodiment is implemented by a computer 1000 having a configuration such as that shown in Figure 16. Figure 16 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the image processing device 10. The computer 1000 has a processing circuitry 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.
[0127] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.
[0128] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.
[0129] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores the program according to this embodiment, which is an example of program data 1450.
[0130] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.
[0131] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.
[0132] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (Organic Electro Luminescence Display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.
[0133] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.
[0134] For example, when the computer 1000 functions as an image processing device 10, the processing circuit 1100 of the computer 1000 realizes the functions of the control unit 12 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the program according to this embodiment and the data in the storage unit 11.
[0135] The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as an alternative, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside the computer 1000, but may be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPUs, MPUs, GPUs, APUs, ASICs, and FPGAs can all be considered integrated circuits.
[0136] <<6. Conclusion>> As described above, according to one embodiment of the present disclosure, the image processing device 10 (corresponding to an example of an "information processing device") includes a control unit 12. The control unit 12 includes a renderer 12b, a phase shift processing unit 12d, a sharpness intensity analysis unit 12h, and a sharpness processing unit 12e. The renderer 12b performs rendering in DNN processing for video having an RNN structure, shifting the phase of the pixel sampling position for each current frame of a series of video frames. The phase shift processing unit 12d performs a phase shift that interpolates the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position. The sharpness intensity analysis unit 12h determines the sharpness intensity of the sharpness processing for the current frame based on the difference in contrast between the history frame, which has motion compensation applied to the DNN output result one frame before the current frame, and the current frame after the phase shift. The sharpness processing unit 12e performs sharpness processing based on the sharpness intensity. This makes it possible to optimize sharpness processing for the current frame.
[0137] Although the embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.
[0138] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.
[0139] Furthermore, this technology can also be configured as follows: (1) A computer-based information processing method comprising: in DNN processing for video having an RNN structure, rendering is performed while shifting the phase of the pixel sampling position for each current frame of a series of video frames; a phase shift is performed to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; the sharpness intensity of the sharpness processing for the current frame is determined based on the difference in contrast between a history frame, which has motion compensation applied to the DNN output result one frame prior to the current frame, and the current frame after the phase shift; and the sharpness processing is performed based on the sharpness intensity. (2) The information processing method according to (1), wherein the sharpness intensity is determined such that the difference in contrast between the history frame and the current frame becomes small. (3) The information processing method according to (1) or (2), comprising: (4) The information processing method according to (3), comprising: (5) The information processing method according to (4), comprising: (6) The information processing method according to (5), comprising: (7) The information processing method according to (5), comprising: (8) The information processing method according to (5), comprising: (9) The information processing method according to (5), comprising: (1(7) The information processing method according to (6), wherein if the pixel of interest is estimated to be the noise-generating pixel, the sharpness intensity of the pixel of interest is changed so that the sharpness processing is suppressed, and if the pixel of interest is estimated not to be the noise-generating pixel, the sharpness intensity of the pixel of interest is not changed. (8) The information processing method according to (7), wherein if the pixel of interest is estimated to be the noise-generating pixel, the sharpness intensity of the pixel of interest is changed to 0. (9) An information processing apparatus comprising: a renderer that renders a series of video frames by shifting the phase of the sampling position of pixels for each current frame in a series of video frames in a DNN processing for video having an RNN structure; a phase shift processing unit that performs a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; a sharpness intensity analysis unit that determines the sharpness intensity of sharpness processing for the current frame based on the difference in contrast between a history frame, which is the DNN output result of the frame one frame prior to the current frame, and the current frame after the phase shift; and a sharpness processing unit that performs the sharpness processing based on the sharpness intensity. (10) The information processing apparatus according to (9), wherein the sharpness intensity analysis unit determines the sharpness intensity such that the difference in contrast between the history frame and the current frame becomes small. (11) The information processing apparatus according to (9) or (10), wherein the sharpness intensity analysis unit calculates the difference in contrast for each local region centered on the pixel of interest in the current frame, and determines the sharpness intensity for each pixel of interest based on the difference in contrast for each local region. (12) The information processing apparatus according to (11), wherein the sharpness intensity analysis unit determines the sharpness intensity for all pixels in the current frame.(13) The information processing apparatus according to (12), wherein the sharpness intensity analysis unit estimates a noise-generating pixel based on the difference in luminance values between the current frame and the history frame in each local region, and the variance value of the luminance values in the local region of the current frame. (14) The information processing apparatus according to (13), wherein the sharpness intensity analysis unit estimates that the pixel of interest is the noise-generating pixel when both the difference in luminance values and the variance value exceed a predetermined threshold. (15) The information processing apparatus according to (14), wherein the sharpness intensity analysis unit changes the sharpness intensity of the pixel of interest to suppress the sharpness processing when the pixel of interest is estimated to be the noise-generating pixel, and does not change the sharpness intensity of the pixel of interest when the pixel of interest is estimated not to be the noise-generating pixel. (16) The information processing apparatus according to (15), wherein the sharpness intensity analysis unit changes the sharpness intensity of the pixel of interest to 0 when the pixel of interest is estimated to be the noise-generating pixel. (17) A computer-readable non-temporary storage medium storing a program that enables a computer to perform the following in a DNN processing for video having an RNN structure: rendering while shifting the phase of the pixel sampling position for each current frame of a series of video frames; performing a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; determining the sharpness intensity of sharpness processing for the current frame based on the difference in contrast between a history frame on which motion compensation has been performed on the DNN output result of the frame before the current frame and the current frame after the phase shift; and performing the sharpness processing based on the sharpness intensity.
[0140] 3 Input Unit 5 Output Unit 10 Image Processing Device 11 Storage Unit 11a DNN Model Information 11b History Information 12 Control Unit 12a Acquisition Unit 12b Renderer 12c Scaler 12d Phase Shift Processing Unit 12e Sharpness Processing Unit 12f High-Quality Image Processing Unit 12fa DNN 12g MC Unit 12h Sharpness Intensity Analysis Unit 12i Output Control Unit
Claims
1. A computer-based information processing method for video having an RNN structure, comprising: rendering while shifting the phase of the pixel sampling position for each current frame of a series of video frames; performing a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; determining the sharpness intensity for sharpness processing on the current frame based on the difference in contrast between a history frame on which motion compensation has been performed on the DNN output result one frame prior to the current frame and the current frame after the phase shift; and performing the sharpness processing based on the sharpness intensity.
2. The information processing method according to claim 1, comprising determining the sharpness intensity such that the difference in contrast between the history frame and the current frame is reduced, with the history frame as the reference.
3. The information processing method according to claim 1, comprising: calculating the difference in contrast for each local region centered on the pixel of interest in the current frame; and determining the sharpness intensity for each pixel of interest based on the difference in contrast for each local region.
4. The information processing method according to claim 3, comprising determining the sharpness intensity for all pixels of the current frame.
5. The information processing method according to claim 4, comprising estimating the noise-generating pixels based on the difference in luminance values between the current frame and the history frame in each local region, and the variance of the luminance values in the local region of the current frame.
6. The information processing method according to claim 5, wherein when both the difference in luminance values and the variance value exceed a predetermined threshold, the pixel of interest is estimated to be the noise-generating pixel.
7. The information processing method according to claim 6, comprising: if the pixel of interest is estimated to be the noise-generating pixel, changing the sharpness intensity of the pixel of interest so that the sharpness processing is suppressed; and if the pixel of interest is estimated not to be the noise-generating pixel, not changing the sharpness intensity of the pixel of interest.
8. The information processing method according to claim 7, further comprising changing the sharpness intensity of the pixel of interest to 0 when it is estimated that the pixel of interest is the noise-generating pixel.
9. An information processing device comprising: a renderer that performs rendering while shifting the phase of the pixel sampling position for each current frame of a series of video frames in a DNN processing for video having an RNN structure; a phase shift processing unit that performs a phase shift to interpolate the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; a sharpness intensity analysis unit that determines the sharpness intensity of sharpness processing for the current frame based on the difference in contrast between a history frame, which has motion compensation applied to the DNN output result one frame prior to the current frame, and the current frame after the phase shift; and a sharpness processing unit that performs the sharpness processing based on the sharpness intensity.
10. The information processing apparatus according to claim 9, wherein the sharpness intensity analysis unit determines the sharpness intensity based on the history frame so that the difference in contrast between it and the current frame is reduced.
11. The information processing apparatus according to claim 9, wherein the sharpness intensity analysis unit calculates the difference in contrast for each local region centered on the pixel of interest in the current frame, and determines the sharpness intensity for each pixel of interest based on the difference in contrast for each local region.
12. The information processing apparatus according to claim 11, wherein the sharpness intensity analysis unit determines the sharpness intensity for all pixels of the current frame.
13. The information processing apparatus according to claim 12, wherein the sharpness intensity analysis unit estimates the noise-generating pixels based on the difference in luminance values between the current frame and the history frame in each local region, and the variance value of the luminance values in the local region of the current frame.
14. The information processing apparatus according to claim 13, wherein the sharpness intensity analysis unit estimates that the pixel of interest is the noise-generating pixel when both the difference in brightness values and the variance value exceed a predetermined threshold.
15. The information processing apparatus according to claim 14, wherein the sharpness intensity analysis unit modifies the sharpness intensity of the pixel of interest to suppress the sharpness processing when it is estimated that the pixel of interest is the noise-generating pixel, and does not modify the sharpness intensity of the pixel of interest when it is estimated that the pixel of interest is not the noise-generating pixel.
16. The information processing apparatus according to claim 15, wherein the sharpness intensity analysis unit changes the sharpness intensity of the pixel of interest to 0 when it is estimated that the pixel of interest is the noise-generating pixel.
17. A computer-readable non-temporary storage medium that stores a program that enables a computer to perform the following in a DNN processing for video having an RNN structure: rendering while shifting the phase of the pixel sampling position for each current frame of a series of video frames; a phase shift that interpolates the pixel value of each pixel in the rendered current frame to the pixel value at the center position of each pixel based on the sampling position; determining the sharpness intensity for sharpness processing on the current frame based on the difference in contrast between a history frame on which motion compensation has been applied to the DNN output result of the frame before the current frame and the current frame after the phase shift; and performing the sharpness processing based on the sharpness intensity.