Information processing device and information processing method

Abstract

This information processing device trains an arbitrary learning model by using, as a training pair, a new viewpoint image generated through three-dimensional reconstruction of a ground truth (GT) image and the GT image, and comprises a control part. The control part detects the positional deviation amount between the new viewpoint image and the GT image for an arbitrary pixel unit in the new viewpoint image and the GT image. The control part performs alignment between the new viewpoint image and the GT image for the pixel unit on the basis of the positional deviation amount. The control part trains the learning model by using, as the training pair, the new viewpoint image and the GT image after alignment.

Description

Information Processing Apparatus and Information Processing Method

[0001] The present disclosure relates to an information processing apparatus and an information processing method.

[0002] Conventionally, as techniques for performing learning-based three-dimensional reconstruction from multi-viewpoint images for each actual viewpoint, methods such as NeRF (Neural Radiance Fields) and "3D Gaussian Splatting" are known. Methods for reconstructing and synthesizing images of novel viewpoints, such as NeRF and "3D Gaussian Splatting", may be referred to as "Novel View Synthesis".

[0003] For example, NeRF is a method that realizes rendering of a 3D scene at an arbitrary viewpoint (novel viewpoint) by training a neural network so that when a group of multi-viewpoint captured images is input, each captured image can be reconstructed into the same image.

[0004] However, in such three-dimensional reconstruction techniques, for example, the rendered images of the 3D scene reconstructed by NeRF include various degradations such as jaggedness, blurring, and floaters, and there is also a problem that the resolution is significantly lower compared to the captured images serving as the teacher.

[0005] Therefore, in order to solve this problem, a method may be adopted in which a super-resolution model is trained using a pair of a rendered image obtained by three-dimensionally reconstructing a captured image and the captured image as a learning pair, and the super-resolution model improves the degradation and enhances the resolution. That is, this method is a method that combines three-dimensional reconstruction technology and super-resolution technology (see, for example, Non-Patent Documents 1 and 2).

[0006] When this method is adopted, for example, in NeRF, since the model is trained to be specialized for a specific scene (hereinafter, appropriately referred to as "scene specialization"), it is also possible to perform specialized super-resolution in which the super-resolution model is trained with scene specialization. It is known that this improves the quality compared to general-purpose super-resolution without scene specialization.

[0007] Wang, Zhongshu, et al. "4k-nerf: High fidelity neural radiance fields at ultra high resolutions." arXiv. 2022. Huang, Xudong, et al. "Refsr-nerf: Towards high fidelity and super resolution view synthesis." CVPR. 2023.

[0008] However, complex positional shifts can occur between the rendering and captured image pairs used for training, as described above. These complex shifts include not only two-dimensional shifts but also three-dimensional shifts. In some cases, one object may be shifted a few pixels to the left while another is shifted a few pixels to the right, resulting in complex positional shifts.

[0009] If the aforementioned rendering and captured image pairs are used directly as training pairs for a super-resolution model while such positional misalignments are occurring, there is a risk of significantly degrading the quality of the trained super-resolution model. This issue is not limited to super-resolution models, but is common to all training models that use the aforementioned rendering and captured image pairs as training pairs.

[0010] Therefore, this disclosure proposes an information processing device and an information processing method that can ensure the quality of the learned model even if a positional shift occurs between pairs of images that form the training pair based on three-dimensional reconstruction.

[0011] To solve the above problems, an information processing device according to this disclosure is an information processing device that trains an arbitrary learning model using a new viewpoint image generated by three-dimensional reconstruction of a GT (Ground Truth) image and the GT image as a training pair, and comprises a control unit. The control unit detects the amount of positional displacement between the new viewpoint image and the GT image for any pixel unit in the new viewpoint image and the GT image. The control unit also performs alignment between the new viewpoint image and the GT image on a pixel-by-pixel basis based on the amount of positional displacement. The control unit also trains the learning model using the new viewpoint image and the GT image after the alignment as the training pair.

[0012] This figure shows the processing flow of 3D reconstruction using NeRF. This figure shows an example of positional misalignment caused by 3D reconstruction. This figure shows a general processing flow when NeRF and super-resolution technology are combined. This is an overview diagram of an information processing method according to an embodiment of this disclosure. This is a block diagram showing an example of the configuration of an information processing device according to an embodiment of this disclosure. This is a flowchart showing the processing procedure executed by an information processing device according to an embodiment of this disclosure. This is a figure (1) showing a specific example of positional misalignment detection processing. This is a figure (2) showing a specific example of positional misalignment detection processing. This is a figure (3) showing a specific example of positional misalignment detection processing. This is a figure (1) showing a specific example of alignment processing. This is a figure (2) showing a specific example of alignment processing. This is a figure (3) showing a specific example of alignment processing. This is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0013] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0014] Furthermore, in the following, the information processing device according to the embodiment of the present disclosure (hereinafter referred to as "this embodiment") is assumed to be an information processing device 100 (see Figure 5) that generates an arbitrary viewpoint image by three-dimensional reconstruction based on multi-view images and generates a super-resolution image by applying the model coefficients of a super-resolution model to the arbitrary viewpoint image.

[0015] Furthermore, in the following, the information processing method according to this embodiment is assumed to be the information processing method executed by the information processing device 100. The information processing method according to this embodiment performs a learning process and a display process. In the learning process, a group of multi-viewpoint captured images and a group of camera poses of the shooting viewpoints of the group of captured images are taken as input, and an image generation model is trained so that a group of three-dimensional reconstructed images corresponding to each shooting viewpoint, the same as the group of captured images, is generated according to each camera pose. In the display process, when camera pose information representing a camera pose of an arbitrary viewpoint is input, an image of the arbitrary viewpoint corresponding to the camera pose information is generated using the image generation model and displayed on a display or the like.

[0016] Furthermore, the following example uses NeRF as a 3D reconstruction technique. The NeRF model is an example of an image generation model. The NeRF model generates a 3D reconstructed image corresponding to an input arbitrary viewpoint (new viewpoint). Note that in the following, "3D reconstruction" may be simply referred to as "reconstruction." The reconstructed image generated by the NeRF model is an example of a "new viewpoint image." A new viewpoint includes the viewpoint from which the captured image was taken.

[0017] Furthermore, in the following, "objects" in an image include not only tangible objects that are captured or drawn within the image, but also intangible elements present in the image, such as the background. Note that the object of interest included in the captured image may sometimes be referred to as the "subject."

[0018] Furthermore, this disclosure will be described in the following order of items: 1. Overview 2. Example of the configuration of an information processing device according to an embodiment of this disclosure 2-1. Block diagram 2-2. Processing procedure of the information processing device according to an embodiment of this disclosure 3. Modifications 4. Hardware configuration 5. Conclusion

[0019] <<1. Overview>> First, an overview of the information processing method according to this embodiment will be explained using Figures 1 to 4. Figure 1 is a diagram showing the processing flow of 3D reconstruction using NeRF. Figure 2 is a diagram showing an example of positional displacement caused by 3D reconstruction. Figure 3 is a diagram showing a general processing flow when NeRF and super-resolution technology are combined. Figure 4 is a schematic diagram illustrating the information processing method according to the embodiment of this disclosure.

[0020] First, prior to describing the information processing method according to the embodiments of this disclosure, we will explain examples of positional misalignment caused by three-dimensional reconstruction, the processing flow of three-dimensional reconstruction using NeRF, and a general processing flow when combining NeRF with super-resolution technology.

[0021] NeRF is a method that enables the rendering of 3D scenes from arbitrary viewpoints by training a neural network to reconstruct the same image from a set of multi-viewpoint captured images.

[0022] Specifically, as shown in Figure 1, NeRF takes a multi-viewpoint captured image set and a set of camera poses for the shooting viewpoints of the captured image set as input, and trains the NeRF model 104b to generate a set of reconstructed images corresponding to each shooting viewpoint, the same as the captured image set, according to each camera pose. The NeRF model 104b is a neural network.

[0023] Here, NeRF trains the NeRF model 104b on a specific 3D scene in which a set of multi-viewpoint images were captured. On the other hand, when NeRF receives camera poses from arbitrary viewpoints, including viewpoints other than the capture viewpoint, it renders an arbitrary viewpoint image corresponding to the camera pose of the arbitrary viewpoint by inference using the trained NeRF model 104b. As a result, a reconstructed image corresponding to the arbitrary viewpoint is generated. In the following, the camera pose of an arbitrary viewpoint may be referred to as "camera pose information".

[0024] However, the reconstructed images of 3D scenes rendered in this way have various degradations, such as jagged edges, blurring, and floaters, and their resolution is significantly lower compared to the captured images.

[0025] Therefore, to resolve this problem, a method is sometimes employed in which a super-resolution model is trained using each pair of captured images and reconstructed images corresponding to the shooting viewpoint shown in Figure 1 as training pairs, and this super-resolution model is used to improve degradation and enhance the perceived resolution. For example, this method combines NeRF and super-resolution technology.

[0026] However, complex positional shifts can occur in the images between the aforementioned training image and reconstructed image pairs. These complex shifts include not only two-dimensional positional shifts but also three-dimensional shifts in the images.

[0027] Let's explain this in detail using Figure 2. In the following, the captured images used in the training pair for the image generation model / image processing model will be referred to as "Ground Truth images (GT images)" as appropriate. GT images are a type of training dataset and are sometimes called training images.

[0028] Figure 2 schematically shows an example where the reconstructed image, which is 3D reconstructed in accordance with the shooting viewpoint, is shifted by about 10 pixels diagonally downward to the right relative to the captured image (GT image). For example, in Figure 2, it can be seen that the lengths of distances i1 and i2 are different between the GT image and the reconstructed image, and that there are differences in the contour lines of objects within the regions R1 and R2 of the GT image and within the corresponding regions of the reconstructed image.

[0029] Such positional shifts are known to occur when performing 3D reconstruction using GT images taken by cameras with lens-induced distortion, or when the camera pose estimation for 3D reconstruction is insufficient and the camera pose cannot be correctly determined. In some cases, complex positional shifts may occur, such as one object being shifted a few pixels to the left while another object is shifted a few pixels to the right.

[0030] If such positional misalignment occurs, and a pair of GT images and reconstructed images, as shown in Figure 2, is used directly as a training pair for a super-resolution model, there is a risk that the quality of the trained super-resolution model will be significantly impaired.

[0031] Let's explain this in more detail, referring to Figure 3. Figure 3 shows a general processing flow when the NeRF model 104b and the super-resolution model 104c are combined.

[0032] The super-resolution model 104c is a neural network that, during display processing, is trained to perform super-resolution by applying super-resolution model coefficients to a reconstructed image that corresponds to an arbitrary viewpoint and has been rendered as an arbitrary viewpoint image.

[0033] As shown in Figure 3, even when NeRF and super-resolution technology are combined, similar to the case in Figure 1, NeRF first takes a group of GT images and a group of camera poses (not shown) of the shooting viewpoints of the GT images as input, and trains the NeRF model 104b so that it can generate a group of reconstructed images corresponding to the camera pose of each shooting viewpoint.

[0034] The reconstructed images corresponding to the camera pose of each shooting viewpoint become the super-resolution learning images used to train the super-resolution model 104c. That is, during the learning process in the example in Figure 3, NeRF performs 3D scene learning and inference (step S1). The reconstructed images corresponding to each shooting viewpoint are examples of "new viewpoint images" as described above.

[0035] Furthermore, during the learning process, the reconstructed image and GT image generated in step S1 are used as a training pair to train the super-resolution model 104c (step S2). The GT image, as one of the training pairs, becomes a training image that shows the correct answer when the other of the training pairs, the reconstructed image, is input to the super-resolution model 104c.

[0036] On the other hand, during display processing using the trained NeRF model 104b, NeRF receives a camera pose from an arbitrary viewpoint and renders the image from that viewpoint (step S3). As a result, a reconstructed image corresponding to the arbitrary viewpoint is generated. Then, the super-resolution model coefficients of the super-resolution model 104c are applied to the reconstructed image corresponding to the arbitrary viewpoint (step S4), and a super-resolution image is generated and output.

[0037] However, in the example shown in Figure 3, if a positional misalignment occurs between the training pair of GT images and the reconstructed image in step S2, the quality of the super-resolution model 104c, and consequently the processing quality in step S4, may be significantly impaired.

[0038] Therefore, in the information processing method according to this embodiment, as shown in Figure 4, the information processing device 100 is equipped with a positional displacement detection unit 105c and a positional alignment processing unit 105d.

[0039] The positional displacement detection unit 105c performs a positional displacement detection process (step S1-2) to detect the amount of positional displacement of objects in each image of the GT image group and each image of the reconstructed image group that were input or output during the 3D scene learning and inference in step S1-1.

[0040] Furthermore, the alignment processing unit 105d performs an alignment process (step S1-3) to align the objects based on the amount of misalignment detected in step S1-2. In the example in Figure 4, the alignment processing unit 105d aligns the GT image with the reconstructed image. As a result of step S1-3 being executed, a group of aligned captured images (GT images) is output. Note that the alignment processing unit 105d may also perform alignment of the reconstructed image with the GT image, not limited to the example in Figure 4.

[0041] Once the aligned GT image set is output, the super-resolution model 104c is trained using this aligned GT image set and the reconstructed image set corresponding to each shooting viewpoint as training pairs (step S2). Subsequently, steps S3 and S4 are executed as in the example in Figure 3.

[0042] According to the information processing method according to this embodiment, even when there is a misalignment between the GT image and the reconstructed image due to three-dimensional reconstruction, alignment can be performed. In addition, since the super-resolution model 104c can be trained with the paired GT image and reconstructed image after alignment as the learning target, the quality of the super-resolution model 104c can be ensured. As a result, since super-resolution can be performed using the super-resolution model 104c with ensured quality, it is possible to improve the degradation generated by NeRF and enhance the resolution feeling.

[0043] Hereinafter, a configuration example of the information processing apparatus 100 to which the information processing method according to this embodiment is applied will be described more specifically.

[0044] <<2. Configuration example of the information processing apparatus according to the embodiment of the present disclosure>> <<2-1. Block diagram>> FIG. 5 is a block diagram showing a configuration example of the information processing apparatus 100 according to the embodiment of the present disclosure. In FIG. 5, only the components necessary for the description of this embodiment are shown as functional blocks, and the description of general components is omitted.

[0045] In addition, in the description using FIG. 5, the description of the components that have already been described may be simplified or omitted.

[0046] As shown in FIG. 5, the information processing apparatus 100 includes an input unit 101, an output unit 102, a communication unit 103, a storage unit 104, and a control unit 105.

[0047] The input unit 101 is a component that inputs data to the information processing apparatus 100. The input unit 101 is realized by an input device such as a keyboard or a mouse, for example, and receives data input by the user. The data input via the input unit 101 is, for example, camera pose information or the like.

[0048] In addition, the input unit 101 is realized as an input interface from various devices, for example. The input unit 101 receives the input of the GT image group at each shooting viewpoint captured by a camera (not shown).

[0049] The output unit 102 is a component that outputs data from the information processing device 100. The output unit 102 includes, for example, a display. The display shows video in which each frame image is a super-resolution image that has been reconstructed and super-resolved using, for example, the NeRF model 104b and the super-resolution model 104c.

[0050] The communication unit 103 is implemented, for example, by a network adapter, and communicates with the external device 200. The external device 200 is a device that performs physical calculations based on data transmitted from the information processing device 100, and is, for example, a video editing device. The communication unit 103 transmits, for example, the video displayed on the display and various information related to this video to the external device 200.

[0051] The memory unit 104 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), or flash memory, or by storage devices such as hard disks or optical discs.

[0052] In the example shown in Figure 5, the storage unit 104 stores the GT image database 104a, the NeRF model 104b, and the super-resolution model 104c. The GT image database 104a is a database that stores the GT image group acquired by the acquisition unit 105a, which will be described later. The GT image database 104a also stores camera parameters (including camera pose) corresponding to each GT image, linked to each GT image.

[0053] Since the NeRF model 104b and the super-resolution model 104c have already been explained, their explanation will be omitted here.

[0054] The control unit 105 corresponds to a so-called processor or controller. The control unit 105 is implemented by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 105 executes the program according to this embodiment (not shown) stored in the storage unit 104, using RAM as the working area. The control unit 105 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0055] The control unit 105 includes an acquisition unit 105a, a NeRF learning unit 105b, a positional misalignment detection unit 105c, a positional alignment processing unit 105d, a super-resolution learning unit 105e, a rendering unit 105f, and a super-resolution unit 105g, and uses the storage unit 104 as appropriate to realize or execute the information processing functions and operations described below.

[0056] The acquisition unit 105a acquires a group of GT images via the input unit 101, which are used for training the NeRF model 104b and the super-resolution model 104c. The acquisition unit 105a also acquires camera pose information via the input unit 101.

[0057] The NeRF learning unit 105b takes a set of multi-viewpoint GT images as input and performs NeRF learning processing to learn the 3D representation (radiance field) of the scene while maintaining three-dimensional consistency. In other words, the NeRF learning unit 105b learns the NeRF model 104b.

[0058] The misalignment detection unit 105c performs the misalignment detection process (step S1-2) described using Figure 4. Specific examples of the misalignment detection process will be described later using Figures 7 to 9. The alignment processing unit 105d performs the alignment process (step S1-3) described using Figure 4. Specific examples of the alignment process will be described later using Figures 10 to 12.

[0059] The super-resolution learning unit 105e performs a super-resolution learning process in which it trains the super-resolution model 104c using pairs of GT images and reconstructed images that have been aligned by the alignment processing unit 105d as training pairs.

[0060] When camera pose information is acquired by the acquisition unit 105a, the rendering unit 105f performs rendering using the NeRF model 104b according to the camera pose information.

[0061] The super-resolution unit 105g performs super-resolution processing on the reconstructed image rendered by the rendering unit 105f, using the super-resolution model 104c. Specifically, the super-resolution unit 105g applies the coefficients of the learned super-resolution model 104c to the reconstructed image rendered by the rendering unit 105f.

[0062] <2-2. Processing Procedure for Information Processing According to the Embodiment of the Present Disclosure> Next, the processing procedure for information processing according to the embodiment of the present disclosure will be described with reference to Figure 6. Figure 6 is a flowchart of the processing procedure executed by the information processing apparatus according to the embodiment of the present disclosure. As shown in Figure 6, steps S101 to S107 correspond to learning processing. Steps S108 to S110 correspond to display processing.

[0063] As shown in Figure 6, in the learning process, the control unit 105 first acquires the GT image group via the input unit 101 using the acquisition unit 105a (step S101). Then, the NeRF learning unit 105b learns the NeRF model 104b (step S102). In step S102, the NeRF learning unit 105b takes the GT image group and the camera pose group of the shooting viewpoint of the GT image group as input and learns the 3D representation (radiance field) of the scene while maintaining three-dimensional consistency.

[0064] Next, the rendering unit 105f generates a set of reconstructed images corresponding to each shooting viewpoint (step S103). NeRF can render arbitrary viewpoint images from any viewpoint based on the learned 3D representation, but when training the super-resolution model 104c in a later stage (step S107 in Figure 6), reconstructed images corresponding to each GT image are generated for training the super-resolution model 104c.

[0065] Next, the positional displacement detection unit 105c detects the amount of positional displacement (step S104). Here, a specific example of the positional displacement detection process performed by the positional displacement detection unit 105c will be explained using Figures 7 to 9.

[0066] Figure 7 is a diagram (part 1) showing a specific example of the positional displacement detection process. Figure 8 is a diagram (part 2) showing a specific example of the positional displacement detection process. Figure 9 is a diagram (part 3) showing a specific example of the positional displacement detection process.

[0067] The geometric shapes, such as ellipses, contained in each image from Figures 7 to 9 schematically represent the objects within each image. Object O1 is an example of a "foreground object." Object O2, which partially overlaps with object O1, is an example of a "background object." This is also true for Figures 10 to 12, which will be shown later.

[0068] First, Figure 7 shows an example in which the positional displacement detection unit 105c detects the amount of positional displacement by feature point-based matching. As shown as points in Figure 7, the positional displacement detection unit 105c detects feature points F (F-1, F-2, F-3, ...) that are common to both the reconstructed image and the GT image, and calculates the positional displacement amounts a1, a2 (a1-1, a1-2, ..., a2-1, ...) of these feature points F.

[0069] Note that feature point F corresponds to an arbitrary pixel unit. A pixel unit includes a first pixel unit and a second pixel unit. For example, if feature point F-2 is the first pixel unit, then feature points F-1 and F-3, which are different from feature point F-2, are the second pixel units. Furthermore, as shown by the difference in the lengths of the positional displacement amounts a1-1, a1-2, and a2-1, the positional displacement amount a1-2 of the first pixel unit is different from the positional displacement amounts a1-1 and a2-1 of the second pixel units.

[0070] Furthermore, the displacement amounts a1 and a2 are vectors, not scalar quantities. For example, if feature points F-1 and F-2 are considered the first pixel unit, then feature point F-3, which is different from feature points F-1 and F-2, is the second pixel unit. And, for example, while displacement amounts a1-1 and a1-2 are shown as rightward arrows, displacement amount a2-1 is shown as a leftward arrow in the opposite direction, the displacement direction of the first pixel unit is different from the displacement direction of the second pixel unit. The complex displacements described above in 3D reconstruction may appear, for example, as opposite displacement directions between a foreground object O1 and a background object O2. Note that the different displacement directions are not limited to opposite directions.

[0071] Examples of feature-point-based matching methods include the KLT (Kanade-Lucas-Tomasi) method, the SIFT (Scale-Invariant Feature Transform) method, the SURF (Speeded-Up Robust Features) method, the ORB (Oriented FAST and Rotated BRIEF) method, and the AKAZE (Accelerated-KAZE) method. Furthermore, feature-point-based methods using neural networks, such as FlowNet, PWC-Net, LiteFlowNet, and SuperPoint, may also be used.

[0072] Furthermore, Figure 8 shows an example in which the positional displacement detection unit 105c detects the positional displacement by one-pixel-based matching on a pixel-by-pixel basis, i.e., on a pixel-by-pixel basis, for each pixel P (P-1, P-2, P-3, ...). As shown in Figure 8, the positional displacement detection unit 105c tracks the difference between each pixel P in the reconstructed image and the GT image and calculates the positional displacement amounts a1-1, a1-2, ..., a2-1, .... Examples of pixel P-based matching methods include the Lucas-Kanade method, Horn-Schunk method, Farneback method, and TV-L1 method.

[0073] Figure 9 also shows an example in which the positional displacement detection unit 105c detects the positional displacement by block-based matching. As shown in Figure 9, the positional displacement detection unit 105c divides the reconstructed image and the GT image into blocks B-1, B-2, B-3… each containing multiple pixels, and calculates the positional displacement amounts a1-1, a1-2, ..., a2-1, ... for each block B-1, B-2, B-3…. Blocks B-1, B-2, B-3… consist of, for example, N×N pixels or N×M pixels (where N and M are natural numbers). For block-based matching, for example, the position of one block of two blocks B is shifted up, down, left, and right by one pixel at a time, and the amount of shift that yields the highest similarity using evaluation metrics such as PSNR (Peak signal-to-noise ratio) or SSIM (Structural SIMilarity) is adopted as the actual positional displacement.

[0074] Returning to the explanation of Figure 6, after step S104 is executed, the misalignment detection unit 105c determines whether or not a misalignment has occurred based on the detection result (step S105). If a misalignment has occurred (step S105, Yes), the alignment processing unit 105d executes the alignment process (step S106). If no misalignment has occurred (step S105, No), the alignment processing unit 105d does not execute the alignment process.

[0075] Here, a specific example of the alignment process performed by the alignment processing unit 105d will be explained using Figures 10 to 12.

[0076] Figure 10 is a diagram (part 1) illustrating a specific example of the alignment process. Figure 11 is a diagram (part 2) illustrating a specific example of the alignment process. Figure 12 is a diagram (part 3) illustrating a specific example of the alignment process.

[0077] First, Figure 10 shows an example in which the alignment processing unit 105d performs warping to align the GT image with the reconstructed image. As shown in Figure 10, the alignment processing unit 105d moves the positions of the feature points F-1, F-2, F-3, ... of the GT image to match the positions of the feature points F-1, F-2, F-3, ... of the reconstructed image, based on the positional displacement amounts a1-1, a1-2, ..., a2-1, ... calculated in step S104. A linear transformation such as homography transformation may also be used. In addition, for areas without feature points F, the pixel value is determined using an arbitrary interpolation process such as bilineal interpolation.

[0078] Furthermore, Figure 11 shows an example in which the alignment processing unit 105d aligns the GT image with the reconstructed image by correcting each region. The misalignment between the GT image and the reconstructed image often occurs due to misalignment of objects O1, O2 within the image, or misalignment of depth.

[0079] Therefore, as shown in Figure 11, the alignment processing unit 105d uses, for example, a depth map or a semantic segmentation map as additional input to acquire regions R3 and R4 of each object O1 and O2, and performs alignment by shifting the position of each of these regions R3 and R4. In this case, as shown in Figure 11, the alignment processing unit 105d performs alignment by correcting each of the regions R3 and R4 in the direction of the arrows in the figure. The regions R3 and R4 shown in Figure 11 correspond to an example of "a region containing an object extracted based on feature points that are commonly included in the reconstructed image (new viewpoint image) and the GT image."

[0080] In this case, the average of the positional displacement amounts of feature points F within regions R3 and R4 may be taken to shift regions R3 and R4 by a certain amount in a certain direction, or, as shown in Figure 9, the positional displacement amount may be calculated by performing a matching process for each block B with respect to regions R3 and R4.

[0081] Furthermore, Figure 12 shows an example in which the alignment processing unit 105d performs alignment of the GT image with the reconstructed image by correcting each block B (B-1, B-2, B-3, ...). Based on the amount of misalignment calculated by block-based matching shown in Figure 9, the alignment processing unit 105d performs a process to shift the position of each block B-1, B-2, B-3, ... in the GT image by a certain amount in a certain direction so that it matches the position of each block B-1, B-2, B-3, ... in the reconstructed image. In this case, as shown in Figure 12, the alignment processing unit 105d performs alignment by correcting each block B-1, B-2, B-3, ... in the direction of the arrow in the figure.

[0082] Returning to the explanation of Figure 6, the super-resolution learning unit 105e then trains the super-resolution model 104c using pairs of GT images and reconstructed images, which have been aligned as needed, as training pairs (step S107).

[0083] Next, in the display process, the acquisition unit 105a acquires camera pose information representing an arbitrary camera pose (step S108). Once the camera pose information is acquired, the rendering unit 105f renders a reconstructed image corresponding to the arbitrary viewpoint according to the acquired camera pose information using the NeRF model 104b (step S109).

[0084] Then, the super-resolution unit 105g applies super-resolution processing to each rendered reconstructed image based on the super-resolution model coefficient (step S110), outputs a super-resolution image, and terminates the processing.

[0085] <<3. Modifications>> Up to this point, embodiments of the present disclosure have been described, but several other modifications of the information processing method according to this embodiment can be given.

[0086] In use cases employing 3D reconstruction technology, it is sometimes necessary to edit multi-view images and then perform 3D reconstruction again. To improve efficiency in this process, the calculated displacement amounts for each image may be output as metadata to a file, rather than recalculating the displacement amounts of objects.

[0087] Furthermore, this embodiment assumes that, in detecting the amount of positional displacement, it is necessary to estimate the amount of positional displacement depending on the learning content. However, in some cases, the amount of positional displacement may not change regardless of the learning content. For example, this may occur when the positional displacement is caused solely by camera lens distortion.

[0088] In this case, even if the learning content changes, the amount of positional shift will not change if the same lens is used for shooting, so it is possible to calculate the amount of positional shift from the lens distortion information. Therefore, instead of calculating the amount of positional shift by comparing the reconstructed image and the GT image, it is also possible to calculate the amount of positional shift using the lens distortion information as input.

[0089] Furthermore, in this embodiment, while examples of alignment processing are mainly given where the GT image is aligned with the reconstructed image, the reconstructed image may also be aligned with the GT image. Also, for example, in the examples shown in Figures 10 to 12, object alignment is performed in all cases, but instead of performing alignment, the objects may be excluded from the training target of the super-resolution model 104c. Specifically, an appropriate threshold may be set for the amount of misalignment in each region calculated by the misalignment detection unit 105c, and regions with misalignment amounts exceeding the threshold may not be used for training. Alternatively, pairs of reconstructed images and GT images that have misalignment amounts exceeding the threshold may be excluded from the training data of the super-resolution model 104c.

[0090] Furthermore, while NeRF was used as an example of 3D reconstruction technology in this embodiment, the 3D reconstruction technology is not limited to NeRF. Any technology capable of reading camera parameters and rendering corresponding arbitrary viewpoint images can be used. For example, "3D Gaussian Splatting" or photogrammetry may be used as the 3D reconstruction technology.

[0091] Furthermore, among the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods.

[0092] In addition, the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each drawing is not limited to the information depicted.

[0093] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0094] For example, the information processing device 100 may be configured as a distributed system consisting of a learning processing device that performs learning processing and a display processing device. In this case, the learning processing device is configured such that the control unit has various processing units, such as an NeRF learning unit 105b that handles the learning processing steps of the NeRF model 104b and the super-resolution model 104c, a positional deviation detection unit 105c, an alignment processing unit 105d, and a super-resolution learning unit 105e. The display processing device is configured such that the control unit has various processing units, such as a rendering unit 105f and a super-resolution unit 105g that handle the display processing steps using the NeRF model 104b and the super-resolution model 104c.

[0095] In this case, the display processing unit will have the trained NeRF model 104b and super-resolution model 104c, which have been trained in the learning processing unit, pre-installed and available for use. Alternatively, the learning processing unit and the display processing unit may be connected via a network, and the trained or updated NeRF model 104b and super-resolution model 104c may be installed in the display processing unit via the network as appropriate.

[0096] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.

[0097] <<4. Hardware Configuration>> Furthermore, the information processing device 100 according to the embodiment of the present disclosure described above is realized by a computer 1000 having a configuration such as that shown in Figure 13. Figure 13 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, and input / output interface 1600. The parts of the computer 1000 are connected by a bus 1050.

[0098] The CPU 1100 operates based on programs stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0099] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0100] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores the program according to this embodiment, which is an example of program data 1450.

[0101] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0102] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0103] For example, when the computer 1000 functions as an information processing device 100, the CPU 1100 of the computer 1000 realizes the functions of the control unit 105 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the program according to this embodiment, various models used in the information processing according to this embodiment, and various data as appropriate. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0104] Furthermore, not limited to the hardware configuration described with reference to Figure 13, the functions realized by the components described herein may be implemented in a circuit configuration or processing circuit configuration, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs, conventional circuits, and / or combinations thereof, programmed to realize the described functions. The processor includes transistors and other circuits and is considered a circuit configuration or processing circuit configuration. The processor may be a programmed processor that executes a program stored in memory. For example, the processor implements the functions of the control unit 105 as a programmed processor.

[0105] In this specification, circuit configurations, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0106] If the hardware is a processor that is considered to be a type of circuit configuration, then the circuit configuration, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0107] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device 100 is an information processing device that trains a super-resolution model 104c (corresponding to an example of an "arbitrary training model") using a new viewpoint image and a GT image generated by three-dimensional reconstruction of a GT image as a training pair, and comprises a control unit 105. The control unit 105 detects the amount of positional displacement between the new viewpoint image and the GT image for each pixel, block, or region (corresponding to an example of an "arbitrary pixel unit") in the new viewpoint image and the GT image. The control unit 105 also performs alignment between the new viewpoint image and the GT image on a pixel-by-pixel basis based on the amount of positional displacement. The control unit 105 also trains the training model using the new viewpoint image and GT image after alignment as a training pair. This ensures that even if a positional displacement occurs between the pair of images that form the training pair based on three-dimensional reconstruction, the quality of the trained super-resolution model 104c can be ensured.

[0108] Although the embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0109] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0110] Furthermore, this technology can also be configured as follows: (1) An information processing device that performs training of an arbitrary learning model using a new viewpoint image generated by three-dimensional reconstruction of a GT (Ground Truth) image and the GT image as a training pair, comprising: a control unit that detects the amount of positional displacement between the new viewpoint image and the GT image for any pixel unit in the new viewpoint image and the GT image; aligns the new viewpoint image and the GT image on a pixel-by-pixel basis based on the amount of positional displacement; and performs training of the learning model using the new viewpoint image and the GT image after the alignment as the training pair. (2) The information processing device according to (1), wherein the control unit performs the alignment by moving one of the pixels of the new viewpoint image or the GT image on a pixel-by-pixel basis based on the amount of positional displacement. (3) The information processing apparatus according to (2), wherein the pixel unit includes a first pixel unit and a second pixel unit, and the control unit detects a first displacement amount of the first pixel unit and a second displacement amount of the second pixel unit that is different from the first displacement amount, and moves the pixels of the first pixel unit and the pixels of the second pixel unit, respectively, based on the first displacement amount and the second displacement amount. (4) The information processing apparatus according to (2), wherein the pixel unit includes a first pixel unit and a second pixel unit, and the control unit moves the pixels of the first pixel unit and the pixels of the second pixel unit, respectively, based on a first displacement direction of the first pixel unit and a second displacement direction of the second pixel unit that is different from the first displacement direction. (5) The information processing apparatus according to (4), wherein the first displacement direction is the opposite direction to the second displacement direction. (6) The information processing apparatus according to (5), wherein the first pixel unit corresponds to a foreground object and the second pixel unit corresponds to a background object. (7) The information processing apparatus according to any one of (2) to (6), wherein the control unit performs the alignment by moving a plurality of pixels of the GT image in pixel units. (8) The information processing apparatus according to any one of (1) to (7), wherein the pixel unit is one pixel.(9) The information processing apparatus according to (8), wherein the one pixel is a pixel corresponding to a feature point that is included in common with the new viewpoint image and the GT image. (10) The information processing apparatus according to any one of (1) to (7), wherein the pixel unit is a region containing an object extracted based on a feature point that is included in common with the new viewpoint image and the GT image. (11) The information processing apparatus according to any one of (1) to (7), wherein the pixel unit is a block containing a plurality of pixels. (12) The information processing apparatus according to any one of (1) to (11), wherein the control unit excludes at least the pixel units that exceed a predetermined threshold from the learning target of the learning model when the detected positional displacement exceeds a predetermined threshold. (13) The information processing apparatus according to any one of (1) to (12), wherein the learning model is a super-resolution model. (14) An information processing method performed by an information processing device that trains an arbitrary learning model using a new viewpoint image generated by three-dimensional reconstruction of a GT (Ground Truth) image and the GT image as a training pair, comprising: detecting a positional displacement between the new viewpoint image and the GT image for any pixel unit in the new viewpoint image and the GT image; performing alignment between the new viewpoint image and the GT image on a pixel-by-pixel basis based on the positional displacement; and training the learning model using the new viewpoint image and the GT image after the alignment as the training pair.

[0111] 100 Information Processing Device 101 Input Unit 102 Output Unit 103 Communication Unit 104 Storage Unit 104a GT Image DB 104b NeRF Model 104c Super Resolution Model 105 Control Unit 105a Acquisition Unit 105b NeRF Learning Unit 105c Positional Shift Detection Unit 105d Alignment Processing Unit 105e Super Resolution Learning Unit 105f Rendering Unit 105g Super Resolution Unit

Claims

1. An information processing device for training an arbitrary learning model using a new viewpoint image generated by three-dimensional reconstruction of a Ground Truth (GT) image and the GT image as a training pair, comprising: a control unit that detects the amount of positional displacement between the new viewpoint image and the GT image for any pixel unit in the new viewpoint image and the GT image; aligns the new viewpoint image and the GT image on a pixel-by-pixel basis based on the amount of positional displacement; and trains the learning model using the newly aligned new viewpoint image and the GT image as the training pair.

2. The information processing apparatus according to claim 1, wherein the control unit performs the alignment by moving one of the pixels of the new viewpoint image or the GT image on a pixel-by-pixel basis based on the amount of positional displacement.

3. The information processing apparatus according to claim 2, wherein the pixel unit includes a first pixel unit and a second pixel unit, the control unit detects a first positional displacement amount of the first pixel unit and a second positional displacement amount of the second pixel unit that is different from the first positional displacement amount, and moves the pixels of the first pixel unit and the pixels of the second pixel unit, respectively, based on the first positional displacement amount and the second positional displacement amount.

4. The information processing apparatus according to claim 2, wherein the pixel unit includes a first pixel unit and a second pixel unit, and the control unit moves the pixels of the first pixel unit and the pixels of the second pixel unit, respectively, based on a first positional displacement direction of the first pixel unit and a second positional displacement direction of the second pixel unit that is different from the first positional displacement direction.

5. The information processing apparatus according to claim 4, wherein the first misalignment direction is in the opposite direction to the second misalignment direction.

6. The information processing apparatus according to claim 5, wherein the first pixel unit corresponds to a foreground object and the second pixel unit corresponds to a background object.

7. The information processing apparatus according to claim 2, wherein the control unit performs the alignment by moving a plurality of pixels of the GT image on a pixel-by-pixel basis.

8. The information processing apparatus according to claim 1, wherein the pixel unit is one pixel.

9. The information processing apparatus according to claim 8, wherein the one pixel is a pixel corresponding to a feature point that is included in common with the new viewpoint image and the GT image.

10. The information processing apparatus according to claim 1, wherein the pixel unit is a region containing an object extracted based on feature points commonly included in the new viewpoint image and the GT image.

11. The information processing apparatus according to claim 1, wherein the pixel unit is a block containing a plurality of pixels.

12. The information processing apparatus according to claim 1, wherein the control unit excludes at least the pixel units exceeding a predetermined threshold from the learning target of the learning model when the detected positional displacement exceeds a predetermined threshold.

13. The information processing apparatus according to claim 1, wherein the learning model is a super-resolution model.

14. An information processing method performed by an information processing device that trains an arbitrary learning model using a new viewpoint image generated by three-dimensional reconstruction of a GT (Ground Truth) image and the GT image as a training pair, comprising: detecting a positional displacement between the new viewpoint image and the GT image for any pixel unit in the new viewpoint image and the GT image; performing alignment between the new viewpoint image and the GT image on a pixel-by-pixel basis based on the positional displacement; and training the learning model using the newly aligned new viewpoint image and the GT image as the training pair.

Images