Training processing device, training processing method, and image generation device

Abstract

This training processing device comprises a control unit. The control unit acquires shape information indicating that a display is a non-planar display. The control unit also generates, by using an image generation model trained to generate a new viewpoint image through three-dimensional reconstruction on the basis of a plurality of ground truth (GT) images respectively corresponding to different imaging viewpoints, at least one new viewpoint image corresponding to a training imaging viewpoint of at least one of the GT images so as to have a non-planar shape corresponding to the non-planar display. Additionally, the control unit generates the GT images having the training imaging viewpoint so as to have a non-planar shape. Furthermore, the control unit trains a super-resolution model for the new viewpoint image by using the new viewpoint image and the GT image each having a non-planar shape as a training pair.

Description

Learning Processing Device, Learning Processing Method, and Image Generation Device

[0001] The present disclosure relates to a learning processing device, a learning processing method, and an image generation device.

[0002] Conventionally, as a technique for performing learning-based three-dimensional reconstruction from multi-viewpoint images for each actual viewpoint, methods based on image generation models such as NeRF (Neural Radiance Fields) and "3D Gaussian Splatting" are known.

[0003] For example, NeRF is a method that realizes the reconstruction of a 3D scene at an arbitrary viewpoint by training a neural network so that when a group of multi-viewpoint captured images is input, the neural network can reconstruct the same image as each captured image.

[0004] As an application destination of such three-dimensional reconstruction technology, there is a case where the rendered reconstructed image is projected not only onto a flat display but also onto a non-flat display such as a curved display or an HMD (Head Mounted Display).

[0005] For example, in-camera VFX (in-camera visual effects) in VP (Virtual Production) that projects a virtual space as a 3D scene onto a large curved display and captures it as a background image of a performer is also one of the uses of three-dimensional reconstruction technology.

[0006] However, in such three-dimensional reconstruction technology, for example, the reconstructed image reconstructed by NeRF includes various degradations such as jaggedness, blurring, and floaters, and there is also a problem that the resolution is significantly lower compared to the captured image serving as the teacher.

[0007] Therefore, in order to solve this problem, a method may be adopted in which a super-resolution model is trained using a pair of a reconstructed image and a captured image as a learning pair, and the super-resolution model improves the degradation and enhances the resolution. That is, this method is a method that combines three-dimensional reconstruction technology and super-resolution technology (see, for example, Non-Patent Documents 1 and 2).

[0008] When this method is adopted, for example, NeRF trains a model specialized for a specific scene (hereinafter referred to as "scene-specific"), and similarly, it is possible to perform specialized super-resolution by training a scene-specific super-resolution model. It is known that this results in improved quality compared to general-purpose super-resolution that is not scene-specific.

[0009] Wang, Zhongshu, et al. "4k-nerf: High fidelity neural radiance fields at ultra high resolutions." arXiv. 2022.Huang, Xudong, et al. "Refsr-nerf: Towards high fidelity and super resolution view synthesis." CVPR. 2023.

[0010] However, when targeting display on a curved display, for example, with the in-camera VFX mentioned above, there is a problem in that applying a scene-specific super-resolution model to the displayed image does not produce the expected effect.

[0011] In front-camera VFX, to prevent distortion from occurring in the image displayed on a curved display when viewing it with the naked eye or photographing it with a camera, a process is sometimes added that deliberately adds distortion to the distortion-free reconstructed image.

[0012] At this point, if the magnification (amount of distortion) of each region differs between the image used to train the super-resolution model and the image to which the super-resolution processing is applied, the result will be that the model is not optimized for the scene.

[0013] Therefore, this disclosure proposes a technology that can improve the degradation of 3D reconstructed images and enhance their resolution, even when the target is display on a non-planar display.

[0014] To solve the above problems, one embodiment of the learning processing device according to the present disclosure includes a control unit. The control unit acquires shape information indicating that the display is a non-planar display. The control unit also uses an image generation model trained to generate new viewpoint images by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint, to generate at least one of the new viewpoint images corresponding to a training shooting viewpoint of at least one of the GT images, such that it has a non-planar shape corresponding to the non-planar display. The control unit also generates the GT image having the training shooting viewpoint so that it has the non-planar shape. Furthermore, the control unit learns a super-resolution model for the new viewpoint images using the new viewpoint images and the GT images, each having the non-planar shape, as a training pair.

[0015] Furthermore, one embodiment of the learning processing method relating to this disclosure includes: acquiring shape information indicating that the display is a non-planar display; generating at least one of the new viewpoint images corresponding to a learning shooting viewpoint of at least one of the GT images, having a non-planar shape corresponding to the non-planar display, using an image generation model trained to generate new viewpoint images by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint; generating the GT image having the learning shooting viewpoint, having the non-planar shape; and training a super-resolution model for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as a training pair.

[0016] Furthermore, one embodiment of the image generation apparatus according to the present disclosure includes a control unit. The control unit acquires shape information of a non-planar display. The control unit also acquires camera pose information representing the camera pose relative to the non-planar display. The control unit generates a new viewpoint image corresponding to the camera pose, based on an image generation model learned to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint, the shape information, and the camera pose information, so that the new viewpoint image has a non-planar shape corresponding to the non-planar display. The control unit also applies super-resolution processing to the generated new viewpoint image having the non-planar shape, based on super-resolution model coefficients learned based on the new viewpoint image having the non-planar shape and the GT image.

[0017] This figure shows the processing flow of 3D reconstruction using NeRF. This figure shows a general processing flow when NeRF and super-resolution technology are combined. This is an overview diagram of an image processing method according to an embodiment of this disclosure. This figure shows an example configuration of an image processing system according to an embodiment of this disclosure. This is a flowchart of the processing procedure executed by the learning processing device according to an embodiment of this disclosure. This is an explanatory diagram (1) of a method for generating a distortion-free super-resolution learning image. This is an explanatory diagram (2) of a method for generating a distortion-inducing super-resolution learning image. This is an explanatory diagram of a method for adding distortion to a GT image. This is a flowchart of the processing procedure executed by an image generation device according to an embodiment of this disclosure. This is a figure (1) showing a modified display. This is a figure (2) showing a modified display. This is a figure (3) showing a modified display. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the learning processing device.

[0018] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0019] Furthermore, the following will use an image processing system 1 (see Figure 4) according to an embodiment of the present disclosure (hereinafter referred to as "this embodiment") as an example. The image processing system 1 performs image processing according to this embodiment. The image processing according to this embodiment performs learning processing and display processing. The learning processing is performed by a learning processing device 100 (see Figure 4) included in the image processing system 1. The display processing is performed by an image generation device 200 (see Figure 4) 5 included in the image processing system 1. In the display processing, the image generation device 200 generates an arbitrary viewpoint image by three-dimensional reconstruction and displays a super-resolution image obtained by applying super-resolution processing to such arbitrary viewpoint image on a non-planar display. In this disclosure, the arbitrary viewpoint image may be referred to as a novel view image.

[0020] Furthermore, the following example uses NeRF as a 3D reconstruction technique. NeRF is an example of an image generation model. Also, in the following, "3D scene" and "3D representation" in expressions such as "learning a 3D scene" and "learned 3D representation" may be replaced with "radiance field" as appropriate. Also, in the following, "3D reconstruction" may sometimes be simply referred to as "reconstruction."

[0021] Furthermore, this disclosure will be described in the following order of items: 1. Overview 2. Example Configuration of an Image Processing System According to an Embodiment of this Disclosure 2-1. Example Configuration of a Learning Processing Device 2-2. Example Configuration of an Image Generation Device 2-3. Processing Procedure for Image Processing According to an Embodiment of this Disclosure 3. Modifications 4. Hardware Configuration 5. Conclusion

[0022] <<1. Overview>> First, an overview of the image processing method according to this embodiment will be explained using Figures 1 to 3. Figure 1 is a diagram showing the processing flow of 3D reconstruction using NeRF. Figure 2 is a diagram showing a general processing flow when NeRF and super-resolution technology are combined. Figure 3 is a schematic diagram illustrating the image processing method according to the embodiment of this disclosure.

[0023] First, before describing the image processing method according to this embodiment, we will explain the processing flow of 3D reconstruction using NeRF, and the general processing flow when combining NeRF with super-resolution technology.

[0024] NeRF is a method that enables the rendering of 3D scenes from arbitrary viewpoints by training a neural network to reconstruct the same image from a set of multi-viewpoint captured images.

[0025] Specifically, as shown in Figure 1, NeRF takes a group of captured images and a group of camera poses corresponding to each shooting viewpoint of the captured images as input, and trains the NeRF model 104b so that it can reconstruct the same group of output images as the captured images according to each camera pose. The NeRF model 104b is a neural network.

[0026] Here, NeRF trains the NeRF model 104b on the specific 3D scene from which the captured images were taken. On the other hand, when camera pose information representing the camera pose of an arbitrary viewpoint is input to NeRF, it renders an arbitrary viewpoint image corresponding to the camera pose of the arbitrary viewpoint by reconstructing it using the trained NeRF model 104b. As a result, the reconstructed image is output.

[0027] However, reconstructed images rendered in this way have various degradations, such as jagged edges, blurring, and floaters, and their resolution is significantly lower compared to the captured image.

[0028] Therefore, to resolve this problem, a method is sometimes employed in which a super-resolution model is trained using each pair of captured images and output images shown in Figure 1 as training pairs, and this super-resolution model is used to improve degradation and enhance the perceived resolution. In other words, this method combines NeRF and super-resolution technology.

[0029] However, even when using this method, there is a problem in that when targeting display on a curved display using in-camera VFX, etc., applying a scene-specific super-resolution model to the displayed image does not produce the expected effect.

[0030] Let's explain this in detail using Figure 2. Figure 2 shows a typical processing flow when combining the NeRF model 104b and the super-resolution model 104c, with the curved display 202a as the display target for an in-camera VFX. In the following, the captured image, which is a Ground Truth image, may be referred to as a "GT image".

[0031] The super-resolution model 104c is a neural network that is trained to perform super-resolution by applying super-resolution model coefficients to a reconstructed image, which is rendered as an arbitrary viewpoint image using the NeRF model 104b, when it is input.

[0032] As shown in Figure 2, even when NeRF is combined with super-resolution technology, NeRF first takes a set of GT images and a set of camera poses representing the viewpoints from which the GT images were taken as input, and trains the NeRF model 104b so that it can reconstruct a set of reconstructed images corresponding to each camera pose.

[0033] The reconstructed image sets corresponding to each camera pose become a group of "super-resolution training images" for training the super-resolution model 104c. That is, in the example in Figure 2, NeRF performs 3D scene training and super-resolution training image generation (step S1). The "super-resolution training image" referred to here corresponds to an example of a "shooting viewpoint image". In this disclosure, a "shooting viewpoint image" may be considered a new viewpoint image generated using the NeRF model 104b so as to correspond to one of the shooting viewpoints of the GT image. Furthermore, the "shooting viewpoints" referenced for training the super-resolution model 104c may each be referred to as "training shooting viewpoints".

[0034] In the example shown in Figure 2, the super-resolution model 104c is trained using the super-resolution training image and GT image generated in step S1 as a training pair (step S2). The GT image, as one of the training pairs, becomes the "ground truth image" when the super-resolution training image, which is the other of the training pairs, is input to the super-resolution model 104c.

[0035] As is also the case in the example in Figure 1, NeRF typically outputs a "super-resolution learning image" that is free from distortion corresponding to the curved surface of the display 202a, as shown by the NeRF model 104b. Naturally, the GT image, which is the training pair for this super-resolution learning image, is also free from distortion. In this disclosure, "distortion" may be referred to as "non-planar" or "non-planar shape."

[0036] On the other hand, when displaying using the trained NeRF model 104b, NeRF receives camera pose information representing the camera pose of an arbitrary viewpoint and renders the image from that arbitrary viewpoint (step S3).

[0037] During this display processing, in the example shown in Figure 2, NeRF outputs a reconstructed image with distortion applied according to the curved surface of the display 202a, based on the display shape information of the display 202a. The display shape information is defined information regarding the shape of the display 202a, such as its size and the curvature of its curved surface. The display shape information is an example of shape information that indicates that the display 202a is a non-planar display.

[0038] Then, during the display processing, in the example shown in Figure 2, the super-resolution model coefficients of the super-resolution model 104c are applied to the reconstructed image output by NeRF (step S4), and a super-resolution image is output. The super-resolution image is then projected onto the display 202a (step S5).

[0039] The steps S3 to S5, which are shown as display processing, are performed by an image generation device 200, for example, located in a studio in an in-camera VFX setup. On the other hand, the learning processing steps S1 to S2 are performed by a learning processing device 100, for example, located in a location other than the studio.

[0040] By the way, in the example shown in Figure 2, as mentioned above, NeRF outputs a reconstructed image with distortion added during display processing (see "Distorted" in the figure). Furthermore, in super-resolution technology, super-resolution is performed on this distorted reconstructed image during display processing, and a distorted super-resolution image is output (see "Distorted" again).

[0041] On the other hand, in the example of FIG. 2, during the learning process, as described above, the super-resolution model 104c is learned using the super-resolution learning image and the GT image without distortion as the learning targets. Therefore, the super-resolution model coefficients of the super-resolution model 104c learned from the super-resolution learning image and the GT image without distortion are applied to the reconstructed image with distortion in step S4. For this reason, in the example of FIG. 2, the quality of the super-resolution process using the super-resolution model 104c is likely to be significantly impaired.

[0042] Therefore, in the image processing method according to the present embodiment, as shown in FIG. 3, when first performing step S1-1, different from the example of FIG. 2, a reconstructed image with distortion corresponding to the curved surface of the display 202a is generated based on the display shape information as the super-resolution learning image. At this time, in the image processing method according to the present embodiment, the reconstructed image with distortion is directly generated from the 三维representation learned in the NeRF model 104b. The specific method will be described later using FIGS. 6 and 7.

[0043] Further, in the image processing method according to the present embodiment, according to the super-resolution learning image with distortion, distortion is also applied to the GT image based on the display shape information (step S1-2). Then, the GT image with distortion is used as the learning target for the super-resolution learning image with the same distortion (see "with distortion" in the figure). The method of applying distortion to the GT image will be described later using FIG. 8.

[0044] Then, the super-resolution model 104c is learned using the super-resolution learning image and the GT image with distortion (step S2), and thereafter, the same steps S3 to S5 as in the example of FIG. 2 are executed. As described above, in the present disclosure, the learning processing apparatus 100 executes steps S1 to S2 which are the learning processes. Also, the image generation apparatus 200 executes steps S3 to S5 which are the display processes.

[0045] According to the image processing method according to this embodiment, the super-resolution model coefficients of the super-resolution model 104c learned using the distortion-free super-resolution learning image and the GT image shown in the example of FIG. 2 as learning targets are not applied to the reconstructed image with distortion during display processing. That is, the quality of the super-resolution processing using the super-resolution model 104c is not significantly impaired.

[0046] Moreover, according to the image processing method according to this embodiment, the amount of distortion of the super-resolution learning image and the GT image, which are the learning targets of super-resolution learning, and the reconstructed image to which the super-resolution model coefficients are applied can be made to match based on the display shape information. As a result, the super-resolution model 104c can be specialized for the NeRF output scene, and it becomes possible to improve the degradation generated by NeRF and enhance the resolution feeling.

[0047] Hereinafter, a configuration example of the image processing system 1 to which the image processing method according to this embodiment is applied will be described more specifically.

[0048] <<2. Configuration example of the image processing system according to the embodiment of the present disclosure>> FIG. 4 is a diagram showing a configuration example of the image processing system 1 according to the embodiment of the present disclosure. In FIG. 4, only the components necessary for the description of this embodiment are shown as functional blocks, and the description of general components is omitted.

[0049] In addition, in the description using FIG. 4, the description of the components that have already been described may be simplified or omitted.

[0050] As shown in FIG. 4, the image processing system 1 includes a learning processing device 100 and an image generation device 200. The learning processing device 100 executes the learning processing of steps S1 to S2 described above. The image generation device 200 executes the display processing of steps S3 to S5 described above.

[0051] <2-1. Configuration example of the learning processing device> The learning processing device 100 includes an input unit 101, an output unit 102, a communication unit 103, a storage unit 104, and a control unit 105.

[0052] The input unit 101 is a component that inputs data to the learning processing device 100. The input unit 101 is implemented by an input device such as a keyboard or mouse, and accepts data input from the user. The data input via the input unit 101 is, for example, display shape information 104d.

[0053] Furthermore, the input unit 101 is implemented, for example, as an input interface from various devices. The input unit 101 receives input of GT images, which are a group of captured images from various real viewpoints taken by a camera (not shown).

[0054] The output unit 102 is a component that outputs data from the learning processing device 100. The output unit 102 includes, for example, a display and a speaker. The output unit 102 outputs visual information, audio information, etc., related to the learning process described above.

[0055] The communication unit 103 is implemented, for example, by a network adapter, and communicates with the image generation device 200.

[0056] The memory unit 104 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), or flash memory, or by storage devices such as hard disks or optical discs.

[0057] In the example shown in Figure 4, the storage unit 104 stores the captured image database 104a, the NeRF model 104b, the super-resolution model 104c, and the display shape information 104d. The captured image database 104a is a database that stores GT images acquired by the acquisition unit 105a, which will be described later. In addition, the captured image database 104a stores camera parameters (including camera pose) corresponding to each GT image, linked to each GT image.

[0058] Since the NeRF model 104b, the super-resolution model 104c, and the display shape information 104d have already been explained, their explanations will be omitted here.

[0059] The control unit 105 corresponds to a so-called processor or controller. The control unit 105 is implemented by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 105 executes the program according to this embodiment (not shown) stored in the storage unit 104, using RAM as the working area. The control unit 105 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0060] The control unit 105 includes an acquisition unit 105a, a NeRF learning unit 105b, a distortion application unit 105c, and a super-resolution learning unit 105d, and while appropriately using the storage unit 104, it realizes or executes the information processing functions and operations described below.

[0061] The acquisition unit 105a acquires GT images via the input unit 101, which are used for training the NeRF model 104b and the super-resolution model 104c. The acquisition unit 105a also acquires display shape information 104d via the input unit 101.

[0062] The NeRF learning unit 105b takes a GT image as input and performs NeRF learning processing to learn the 3D representation (radiance field) of the scene while maintaining three-dimensional consistency. In other words, the NeRF learning unit 105b learns the NeRF model 104b. The NeRF learning unit 105b also transmits the learned NeRF model 104b to the image generation device 200 via the communication unit 103.

[0063] The distortion application unit 105c performs a distortion application process as a preprocessing step for the super-resolution learning process that learns the super-resolution model 104c. This process applies distortion to both the reconstructed images (super-resolution learning images) and the GT images (ground truth images), which are reconstructed using the learned NeRF model 104b and the same images as each GT image. Based on the display shape information 104d, the distortion application unit 105c applies distortion according to, for example, the curvature of the display 202a that is the target of the display.

[0064] The super-resolution learning unit 105d performs a super-resolution learning process to train the super-resolution model 104c using the distorted super-resolution learning image and GT image as training data. The super-resolution learning unit 105d also transmits the trained super-resolution model 104c to the image generation device 200 via the communication unit 103.

[0065] <2-2. Example of Image Generation Device Configuration> Next, the image generation device 200 has an input unit 201, an output unit 202, a communication unit 203, a storage unit 204, and a control unit 205.

[0066] The input unit 201 is a component that inputs data to the image generation device 200. Similar to the input unit 101 described above, the input unit 201 is implemented by an input device such as a keyboard or mouse, and accepts data input from the user. The input unit 201 can also be implemented as an input interface from various devices, such as the camera of an in-camera VFX. The data input via the input unit 201 is, for example, camera pose information that represents the camera pose of an arbitrary viewpoint. The camera pose information represents, for example, the camera position of an in-camera VFX.

[0067] The output unit 202 is a component that outputs data from the image generation device 200. The output unit 202 includes, for example, the display 202a described above. The display 202a displays video in which each frame image is a super-resolution image generated in the image generation device 200 using the NeRF model 104b and the super-resolution model 104c.

[0068] The communication unit 203, like the communication unit 103 described above, is implemented by, for example, a network adapter and communicates with the learning processing unit 100 or the external device 300. The external device 300 is a device that performs physical calculations based on data transmitted from the image generation device 200, and is, for example, an editing device in in-camera VFX. The communication unit 203 transmits, for example, the video displayed on the display 202a and various information related to this video to the external device 300.

[0069] The storage unit 204, like the storage unit 104 described above, is implemented by, for example, semiconductor memory elements such as RAM, ROM, or flash memory, or by a storage device such as a hard disk or optical disc.

[0070] In the example shown in Figure 4, the memory unit 204 stores the NeRF model 104b, the super-resolution model 104c, and the display shape information 104d. The NeRF model 104b and the super-resolution model 104c are image generation models that have been trained in the learning processing unit 100. The NeRF model 104b, the super-resolution model 104c, and the display shape information 104d stored in the memory unit 204 are acquired from the learning processing unit 100 by the acquisition unit 205a, which will be described later.

[0071] Furthermore, the NeRF model 104b, the super-resolution model 104c, and the display shape information 104d may be stored in an external storage device or storage server and shared between the learning processing device 100 and the image generation device 200.

[0072] The control unit 205, like the control unit 105 described above, corresponds to a so-called processor or controller. The control unit 205 is implemented by, for example, a CPU, MPU, GPU, etc. The control unit 205 executes the program according to this embodiment (not shown), stored in the storage unit 204, using RAM as its working area. The control unit 205 can also be implemented by, for example, an integrated circuit such as an ASIC or FPGA.

[0073] The control unit 205 includes an acquisition unit 205a, a rendering unit 205b, a super-resolution unit 205c, and an output control unit 205d, and, while appropriately using the storage unit 204, realizes or executes the information processing functions and operations described below.

[0074] The acquisition unit 205a acquires the NeRF model 104b, the super-resolution model 104c, and the display shape information 104d from the learning processing device 100 via the communication unit 203. Alternatively, the acquisition unit 205a may acquire this data from the learning processing device 100 via a storage medium or the like, without going through the communication unit 203.

[0075] Furthermore, the acquisition unit 205a acquires camera pose information via the input unit 201 when performing display processing to project onto the display 202a.

[0076] When camera pose information is acquired by the acquisition unit 205a, the rendering unit 205b uses the NeRF model 104b to perform rendering according to the camera pose information. In the case of in-camera VFX, etc., in order to display the 3D scene learned by the NeRF model 104b on the display 202a without distortion, the rendering unit 205b performs rendering with distortion based on the display shape information 104d.

[0077] The super-resolution unit 205c performs super-resolution processing on the reconstructed image rendered by the rendering unit 205b, using the super-resolution model 104c. Specifically, the super-resolution unit 205c applies the coefficients of the trained super-resolution model 104c to the reconstructed image rendered by the rendering unit 205b.

[0078] The output control unit 205d performs output control to output the super-resolution image processed by the super-resolution unit 205c to the display 202a.

[0079] <2-3. Image Processing Procedure According to the Embodiment of the Present Disclosure> Next, the image processing procedure according to the embodiment of the present disclosure will be described using Figures 5 to 9. Figure 5 is a flowchart showing the processing procedure executed by the learning processing device 100 according to the embodiment of the present disclosure. Figure 6 is an explanatory diagram of the method for generating a distortion-free super-resolution learning image. Figure 7 is an explanatory diagram of the method for generating a distortion-inducing super-resolution learning image.

[0080] Figure 6 is an explanatory diagram illustrating the method for generating a distortion-free super-resolution learning image, i.e., a reconstructed image by rendering using a normal NeRF, as a comparative example of this embodiment. Figure 8 is an explanatory diagram illustrating the method for adding distortion to a GT image. Figure 9 is a flowchart showing the processing procedure executed by the image generation apparatus 200 according to the embodiment of this disclosure.

[0081] In the image processing according to this embodiment, a learning process and a display process are performed as described above. In this disclosure, the control unit 105 of the learning processing device 100 performs the learning process, and the control unit 205 of the image generation device 200 performs the display process.

[0082] As shown in Figure 5, in the learning process, the control unit 105 of the learning processing device 100 first acquires a group of GT images via the input unit 101 through the acquisition unit 105a (step S101). Then, the NeRF learning unit 105b learns the NeRF model 104b (step S102). In step S102, the NeRF learning unit 105b learns the 3D representation (radiance field) of the scene in order to maintain three-dimensional consistency.

[0083] Then, the acquisition unit 105a determines whether or not the display shape information 104d has been acquired (step S103). If the display shape information 104d has been acquired (step S103, Yes), the distortion application unit 105c generates a super-resolution learning image with distortion (step S104). On the other hand, if the display shape information 104d has not been acquired (step S103, No), the distortion application unit 105c generates a super-resolution learning image without distortion (step S106).

[0084] NeRF can render new viewpoint images from any viewpoint based on the learned 3D representation. However, when training the super-resolution model 104c in a later stage of the training process (step S107 in Figure 5), it generates super-resolution training images, which are new viewpoint images corresponding to the training shooting viewpoints of each GT image.

[0085] The distortion application unit 105c takes camera parameters representing the learning shooting viewpoint for each GT image as input and generates a reconstructed image of the GT image by NeRF corresponding to these camera parameters as a super-resolution learning image. When the NeRF model 104b and the super-resolution model 104c are combined, a reconstructed image with the same amount of distortion as the GT image is usually generated. However, the distortion application unit 105c directly generates a reconstructed image with distortion corresponding to the display shape of the display 202a from the learned 3D representation.

[0086] Specifically, if step S106, i.e., the display shape information 104d, has not been acquired, the distortion application unit 105c, as shown in Figure 6, casts a light ray to each pixel position P1 of the GT image and determines the pixel value of the corresponding pixel through a neural network. In NeRF, the learned 3D representation, i.e., the radiance field in the NeRF model 104b, has coefficients that indicate at least the color and density at each sampling point of the light ray, and the pixel value is determined based on these.

[0087] On the other hand, when step S104, i.e., when display shape information 104d is acquired, the distortion application unit 105c, as shown in Figure 7, sends light rays to each pixel position P2 on the display surface of the display 202a onto which the image is projected, and determines the pixel value of the corresponding pixel through the neural network. As a result, a super-resolution learning image with added distortion, i.e., a non-planar shape corresponding to the display 202a, can be directly rendered and generated from the 3D representation learned in NeRF.

[0088] Then, the distortion application unit 105c provides the super-resolution learning unit 105d with the super-resolution learning image generated in step S104 or step S106, along with the GT image, as training data in step S107. After executing step S104, the distortion application unit 105c further applies distortion to the GT image (step S105).

[0089] The training data for the super-resolution model 104c utilizes the same GT images that were used for training the NeRF model 104b. When the display is a flat-panel display, the GT images are normally used as the ground truth images in the training data for the super-resolution model 104c without adding any distortion. However, in this embodiment, distortion corresponding to the display shape of the display 202a is added to the GT images, and these are used together with the super-resolution training images as the ground truth images in the training data for the super-resolution model 104c in the subsequent step (step S107).

[0090] Specifically, as shown in Figure 8, the distortion-applying unit 105c, similar to the super-resolution learning image described above, emits a light ray to each pixel position P2 on the display surface of the display 202a onto which the image is projected. If the light ray intersects the GT image at pixel position P1, the pixel value of pixel position P1 is substituted for the pixel corresponding to pixel position P2. If the light ray does not intersect the GT image, a black pixel is substituted for the pixel corresponding to pixel position P2. This allows distortion to be applied to the GT image.

[0091] Returning to the explanation of Figure 5, the super-resolution learning unit 105d then trains the super-resolution model 104c using the distorted super-resolution learning image generated in step S104 and the GT image to which distortion was added in step S105 as training data (step S107). Alternatively, in step S107, the super-resolution learning unit 105d trains the super-resolution model 104c using the undistorted super-resolution learning image generated in step S106 and the undistorted GT image as training data.

[0092] Furthermore, if distortion is applied to the GT image in this training data, areas not included in the GT image alone will be rendered as black pixels. Therefore, the super-resolution learning unit 105d additionally extracts these black pixel areas and excludes them from the areas to be trained by the super-resolution model 104c. At this time, instead of excluding the black pixel areas from the areas to be trained, the super-resolution learning unit 105d may change the corresponding areas in the super-resolution training image to black pixels and use them as training data. After the execution of step S107, the control unit 105 of the learning processing device 100 terminates the learning process.

[0093] Next, the processing procedure for display will be explained. Note that the NeRF model 104b and the super-resolution model 104c, which have been trained through the learning process, are pre-installed in the storage unit 204 of the image generation device 200.

[0094] In the display process, as shown in Figure 9, the control unit 205 of the image generation device 200 acquires camera pose information representing an arbitrary camera pose when projecting onto the display 202a (step S201). Then, the rendering unit 205b renders an image of the viewpoint corresponding to the acquired camera pose information using the NeRF model 104b (step S202).

[0095] Then, the super-resolution unit 205c applies super-resolution processing to each reconstructed image rendered by the rendering unit 205b based on the coefficients of the trained super-resolution model 104c (step S203). Then, the output control unit 205d displays the super-resolution images after super-resolution processing as individual frame images on the display 202a (step S204).

[0096] As described above, in the display processing according to this embodiment, when the display 202a is a curved display such as an in-camera VFX, the rendering unit 205b performs rendering with added distortion in order to display the 3D scene learned by the NeRF model 104b on the display 202a without distortion (see Figure 3). Then, the super-resolution unit 205c applies super-resolution processing to each reconstructed image rendered by the rendering unit 205b based on the coefficients of the super-resolution model 104c. This improves the degradation that occurs with NeRF alone and enhances the sense of resolution, enabling the display of high-quality images on the display 202a.

[0097] <<3. Modifications>> While embodiments of the present disclosure have been described, several other modifications of the image processing method according to this embodiment can be given.

[0098] Figure 10 shows a modified display (part 1). Figure 11 shows a modified display (part 2). Figure 12 shows a modified display (part 3).

[0099] The camera 3 shown in Figures 10 to 12 schematically represents the shooting position when photographing the display, or the viewpoint position when viewing the display with the naked eye, or in general, the position from which the display is viewed. The dashed arrows extending from camera 3 indicate the viewing direction for the display.

[0100] In this embodiment, the display 202a that is the target of the display is, for example, a curved display, but the display that is the target of the display may be any non-planar display other than a curved display.

[0101] For example, the display to be used for the display may be a combination of multiple (in this case, two) flat displays, such as displays 202b-1 and 202b-2, as shown in Figure 10. In other words, a display 202b with a curved display surface may be used.

[0102] Furthermore, as shown in Figure 11, this embodiment can also be applied to the HMD 202c, as viewing the flat-panel display 202ca through the lens 202cb causes the display surface to appear distorted.

[0103] Furthermore, as shown in Figure 12, this embodiment can also be applied when the flat-panel display 202b is viewed from an angled direction.

[0104] In other words, in this embodiment, a "non-planar display" refers to a display having an image projection surface that is not perpendicular to the viewing direction. The example in Figure 11 can be said to have an image projection surface that is indirectly not perpendicular to the viewing direction due to the lens 202cb.

[0105] In addition to the method of adding distortion to each training data pair of the super-resolution model 104c as in this embodiment, it is also conceivable that the super-resolution model 104c be trained without adding distortion, and then distortion is added after the super-resolution processing using the super-resolution model 104c for display on a non-planar display.

[0106] In this case, the GT image as is is used as one of the training pairs, and the reconstructed image rendered by a normal NeRF as shown in Figure 6 is used as the other of the training pairs, and the super-resolution model 104c is trained. This has the advantage that display shape information 104d is not required. However, in this case, it is necessary to render arbitrary viewpoint images or apply super-resolution processing using the super-resolution model 104c to areas that are not displayed largely on the display, so there is a concern that the processing efficiency will be poor. In addition, since distortion is applied to the image after super-resolution processing, there is a concern that the image quality will deteriorate. For this reason, this embodiment assumes that super-resolution processing is applied to the reconstructed image after distortion has been applied.

[0107] Furthermore, while NeRF was used as an example of 3D reconstruction technology in this embodiment, the 3D reconstruction technology is not limited to NeRF. Any technology capable of reading camera parameters and rendering corresponding arbitrary viewpoint images can be used. For example, "3D Gaussian Splatting" or photogrammetry may be used as the 3D reconstruction technology.

[0108] Furthermore, the generation of distorted super-resolution learning images does not necessarily have to be done directly from the learned 3D representation. Instead, similar to the distortion applied to GT images, distortion can be applied to the undistorted image. However, since this increases the interpolation process and may slightly reduce quality, the method shown in Figure 7 is preferable.

[0109] Furthermore, among the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods.

[0110] In addition, the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each drawing is not limited to the information depicted.

[0111] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0112] For example, in the embodiment described above, the learning processing device 100 and the image generation device 200 are shown as separate processing devices, but they may also be configured as an integrated image processing device.

[0113] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.

[0114] <<4. Hardware Configuration>> Furthermore, the learning processing device 100 and image generation device 200 according to the embodiments of this disclosure described above are implemented by a computer 1000 having a configuration such as that shown in Figure 13. The learning processing device 100 will be explained as an example. Figure 13 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the learning processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, and input / output interface 1600. The parts of the computer 1000 are connected by a bus 1050.

[0115] The CPU 1100 operates based on programs stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0116] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0117] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores the program according to this embodiment, which is an example of program data 1450.

[0118] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0119] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0120] For example, when the computer 1000 functions as a learning processing unit 100, the CPU 1100 of the computer 1000 realizes the functions of the control unit 105 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the program according to this embodiment, various models used in the information processing according to this embodiment, and various data as appropriate. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0121] Furthermore, not limited to the hardware configuration described with reference to Figure 13, the functions realized by the components described herein may be implemented in a circuit configuration or processing circuit configuration, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs, conventional circuits, and / or combinations thereof, programmed to realize the described functions. The processor includes transistors and other circuits and is considered a circuit configuration or processing circuit configuration. The processor may be a programmed processor that executes a program stored in memory. For example, the processor implements the functions of the control unit 105 as a programmed processor.

[0122] In this specification, circuit configurations, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0123] If the hardware is a processor that is considered to be a type of circuit configuration, then the circuit configuration, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0124] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the learning processing device 100 includes a control unit 105. The control unit 105 acquires display shape information 104d indicating that the display is a non-planar display. The control unit 105 also uses a NeRF model 104b, which has been trained to generate new viewpoint images by three-dimensional reconstruction based on a plurality of GT images, each corresponding to a different shooting viewpoint, to generate at least one of the new viewpoint images corresponding to a learning shooting viewpoint of at least one of the GT images, so as to have a non-planar shape corresponding to the non-planar display. The control unit 105 also generates the GT image having the learning shooting viewpoint so as to have the non-planar shape. The control unit 105 also trains a super-resolution model 104c for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as a training pair. This makes it possible to improve the degradation of the three-dimensionally reconstructed image and enhance the sense of resolution, even when the target is display on a non-planar display.

[0125] Although the embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0126] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0127] Furthermore, this technology can also be configured as follows: (1) A learning processing device comprising: a control unit that acquires shape information indicating that the display is a non-planar display, generates at least one of the new viewpoint images corresponding to a training shooting viewpoint of at least one of the GT images, having a non-planar shape corresponding to the non-planar display, using an image generation model that has been trained to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint, and trains a super-resolution model for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as training pairs. (2) The learning processing device according to (1), wherein the image generation model includes NeRF (Neural Radiance Fields). (3) The learning processing apparatus according to (2), wherein the control unit learns the radiance field of the NeRF based on the plurality of GT images, and generates the GT image having a non-planar shape by shining a ray of light on the GT image located between the learning shooting viewpoint and the non-planar display in the learned radiance field. (4) The learning processing apparatus according to (3), wherein if the ray of light has an intersection with the GT image, the control unit substitutes the pixel value of the intersection point into the pixel position where the ray of light intersects with the non-planar display, and if the ray of light does not intersect with the GT image, the control unit sets the pixel position where the ray of light intersects with the non-planar display as a black pixel. (5) The learning processing apparatus according to (4), wherein the control unit excludes the region of the black pixel from the region of the GT image that is the target of learning the super-resolution model. (6) The learning processing apparatus according to any one of (1) to (5), wherein the shape information represents a curved display. (7) The learning processing device according to any one of (1) to (6) above, wherein the shape information is a parameter for in-camera VFX.(8) The learning processing apparatus according to any one of (1) to (7), wherein the control unit inputs the learning shooting viewpoint and the shape information to the image generation model to reconstruct the new viewpoint image having the non-planar shape. (9) The learning processing apparatus according to any one of (1) to (8), wherein the control unit applies distortion according to the shape information to the new viewpoint image reconstructed in a plane. (10) A learning processing method comprising: acquiring shape information indicating that the display is a non-planar display; generating at least one new viewpoint image corresponding to a learning shooting viewpoint of at least one GT image, having a non-planar shape corresponding to the non-planar display, using an image generation model trained to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint; generating the GT image having the learning shooting viewpoint, having the non-planar shape; and training a super-resolution model for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as a training pair. (11) An image generation apparatus comprising: an image generation model that acquires shape information of a non-planar display, an image generation model that has been trained to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images corresponding to different shooting viewpoints, the shape information, and the camera pose information, which generates the new viewpoint image corresponding to the camera pose so as to have a non-planar shape corresponding to the non-planar display, and which applies super-resolution processing to the generated new viewpoint image having the non-planar shape based on the new viewpoint image having the non-planar shape and the GT image, respectively. (12) The image generation apparatus according to (11), wherein the control unit outputs the new viewpoint image having the non-planar shape to which the super-resolution processing has been applied to the non-planar display.(13) The image generation apparatus according to (11) or (12), wherein the camera pose represents the camera position of an in-camera VFX that photographs the non-planar display.

[0128] 100 Learning Processing Device 101 Input Unit 102 Output Unit 103 Communication Unit 104 Storage Unit 104a Image Capture DB 104b NeRF Model 104c Super Resolution Model 104d Display Shape Information 105 Control Unit 105a Acquisition Unit 105b NeRF Learning Unit 105c Distortion Application Unit 105d Super Resolution Learning Unit 200 Image Generation Device 201 Input Unit 202 Output Unit 203 Communication Unit 204 Storage Unit 205 Control Unit 205a Acquisition Unit 205b Rendering Unit 205c Super Resolution Unit 205d Output Control Unit

Claims

1. A learning processing device comprising: a control unit that acquires shape information indicating that the display is a non-planar display; uses an image generation model trained to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint, to generate at least one of the new viewpoint images corresponding to a training shooting viewpoint of at least one of the GT images, such that it has a non-planar shape corresponding to the non-planar display; generates the GT image having the training shooting viewpoint so that it has the non-planar shape; and learns a super-resolution model for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as a training pair.

2. The learning processing apparatus according to claim 1, wherein the image generation model includes NeRF (Neural Radiance Fields).

3. The learning apparatus according to claim 2, wherein the control unit learns the radiance field of the NeRF based on the plurality of GT images, and generates the GT image having a non-planar shape by shining a light ray on the GT image located between the learning shooting viewpoint and the non-planar display in the learned radiance field.

4. The learning processing apparatus according to claim 3, wherein the control unit, if the light ray has an intersection point with the GT image, substitutes the pixel value of the intersection point into the pixel position where the light ray intersects the non-planar display, and if the light ray does not intersect the GT image, sets the pixel position where the light ray intersects the non-planar display as a black pixel, thereby generating the GT image having a non-planar shape.

5. The learning processing apparatus according to claim 4, wherein the control unit excludes the region of the black pixels from the region of the GT image that is the target of training the super-resolution model.

6. The learning processing apparatus according to claim 1, wherein the shape information represents a curved display.

7. The learning processing apparatus according to claim 1, wherein the shape information is a parameter for an in-camera VFX.

8. The learning processing apparatus according to claim 1, wherein the control unit reconstructs the new viewpoint image having a non-planar shape by inputting the learning shooting viewpoint and the shape information to the image generation model.

9. The learning processing apparatus according to claim 1, wherein the control unit applies a distortion corresponding to the shape information to the new viewpoint image reconstructed on a plane.

10. A learning process method comprising: acquiring shape information indicating that the display is a non-planar display; generating at least one of the new viewpoint images corresponding to a training shooting viewpoint of at least one of the GT images, using an image generation model trained to generate new viewpoint images by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images, each corresponding to a different shooting viewpoint, so as to have a non-planar shape corresponding to the non-planar display; generating the GT image having the training shooting viewpoint so as to have the non-planar shape; and training a super-resolution model for the new viewpoint image using the new viewpoint image and the GT image, each having the non-planar shape, as training pairs.

11. An image generation apparatus comprising: an image generation model that acquires shape information of a non-planar display, an image generation model that has been trained to generate a new viewpoint image by three-dimensional reconstruction based on a plurality of GT (Ground Truth) images corresponding to different shooting viewpoints, the shape information, and the camera pose information, which generates the new viewpoint image corresponding to the camera pose so as to have a non-planar shape corresponding to the non-planar display, and a control unit that applies super-resolution processing to the generated new viewpoint image having the non-planar shape based on the new viewpoint image having the non-planar shape and the GT image, respectively.

12. The image generation apparatus according to claim 11, wherein the control unit outputs the new viewpoint image having a non-planar shape to which the super-resolution processing has been applied to the non-planar display.

13. The image generation apparatus according to claim 11, wherein the camera pose represents the camera position of an in-camera VFX that captures the non-planar display.

Images