Free viewpoint video generation method, information processing device, learning device and program

Abstract

A free viewpoint video generation method executed by an information processing device performs training with video data obtained by imaging an object from a plurality of viewpoints to generate information corresponding to an image from an arbitrary viewpoint at a trained time, and deforms the generated information in a time direction by using a mechanism for predicting deformation information at an arbitrary time, thereby generating an image from the arbitrary viewpoint at the arbitrary time.

Description

Free-viewpoint video generation method, information processing device, learning device, and program

[0001] This invention relates to a technology that generates images viewed from any viewpoint using video data captured from multiple viewpoints.

[0002] Neural Radiance Fields (NeRF) (Non-Patent Document 1) is attracting attention as a technique for generating images from arbitrary viewpoints using a model trained on images taken from multiple viewpoints.

[0003] Furthermore, free-viewpoint video, which can be viewed from any viewpoint, is attracting attention as a type of XR content. Various technologies have been proposed to handle dynamic scenes by extending NeRF in the time direction (for example, K-Planes in Non-Patent Document 2).

[0004] B. Mildenhall, et al., “NeRF: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, 65(1):99-106, 2021. S. Fridovich-Keil, et al., “K-Planes: Explicit Radiance Fields in Space, Time, and Appearance,” in CVPR 2023, 2023. F. Reda, et al., “Film: Frame interpolation for large motion,” in ECCV 2022, 2022.Z. Huang, et al., “Real-Time Intermediate Flow Estimation for Video Frame Interpolation,” in ECCV 2022, 2022.Z. Teed and J. Deng, “RAFT: Recurrent all-pairs field transforms for optical flow,” in Proc. ECCV, 2020.

[0005] However, conventional techniques that extend NeRF in the time direction to support dynamic scenes (for example, Non-Patent Document 2) are designed on the premise that inference is performed on known time points during training, and have the problem that image quality deteriorates when inference is performed on unknown time points.

[0006] This invention has been made in view of the above points, and aims to provide a technology that enables the generation of video of any time viewed from any viewpoint with high accuracy using video data captured from multiple viewpoints.

[0007] According to the disclosed technology, a free-viewpoint video generation method is provided which is executed by an information processing device, wherein the device learns video data obtained by photographing an object from multiple viewpoints, generates information corresponding to an image from an arbitrary viewpoint at a learned time, and then deforms the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time, thereby generating an image from an arbitrary viewpoint at an arbitrary time.

[0008] According to the disclosed technology, a technology is provided that enables the generation of highly accurate video footage of any point in time viewed from any viewpoint, using video data captured from multiple viewpoints.

[0009] This is a diagram showing an example configuration of the learning device 100. This is a flowchart explaining the operation of the learning device 100. This is a diagram showing an example configuration of the inference device 200. This is a flowchart explaining the operation of the inference device 200. This is a diagram showing an example configuration of the information processing device 400. This is a diagram showing an overview of NeRF. This is a diagram explaining NeRF. This is a diagram explaining NeRF. This is a diagram showing an overview of Dynamic NeRF. This is a diagram showing a method for obtaining feature vectors in K-Planes. This is a diagram showing an overview of RIFE. This is a diagram showing an example configuration of the learning unit 120. This is a diagram explaining the operation of the learning unit 120. This is a diagram showing the configuration of the model 300 in the second embodiment. This is a diagram explaining the operation of the model 300 in the second embodiment. This is a diagram showing an image of loss calculation in the learning of the model 300 in the second embodiment. This is a diagram showing an example of the hardware configuration of the device.

[0010] Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. Below, the first embodiment and the second embodiment will be described.

[0011] [First Embodiment]

[0012] (Example of device configuration) The following describes an example of the device configuration in this embodiment. In this embodiment, separate devices are used for the learning phase and the inference phase, but the same device may be used for both the learning phase and the inference phase.

[0013] <Device Configuration in the Learning Phase> Figure 1 shows an example of the configuration of the learning device 100. As shown in Figure 1, the learning device 100 comprises an input unit 110, a learning unit 120, and an output unit 130. The learning unit 120 holds the model 300 to be learned. The model 300 is a model that is learned to take an arbitrary viewpoint and arbitrary time as input and output an image corresponding to that viewpoint and time, based on the learning data (ground truth data). The model 300 is, for example, a neural network model. A detailed example of the model 300 will be described later. The learning device may also be called an information processing device.

[0014] The operation of the learning device 100 will be explained according to the steps in the flowchart in Figure 2. In S101, learning data is input from the input unit 110. The learning data is a collection of camera images (frames) of the target object taken from various directions.

[0015] In S102, the learning unit 120 trains the model 300. Training is performed, for example, by updating the parameters of the model 300 so that the LOSS (loss) based on the error between the image generated by the model 300 and the ground truth image is minimized. Details of the training method will be described later. When training is complete, in S103, the output unit 130 outputs the trained model 300 (specifically, model parameters, hyperparameters, etc.). The trained model 300 is used in the inference phase.

[0016] <Device Configuration in the Inference Phase> Figure 3 shows an example of the configuration of the inference device 200. As shown in Figure 3, the inference device 200 comprises an input unit 210, an inference unit 220, and an output unit 230. The inference unit 220 holds a trained model 300. It can also be considered that the inference unit 220 is the model 300. The trained model 300 is the model 300 trained by the training device 100 described above. The inference device may also be called an information processing device.

[0017] The operation of the inference device 200 will be explained according to the steps in the flowchart of Figure 4. In S201, input data (e.g., viewpoint, time) is input from the input unit 210. In S202, the inference unit 220 inputs the input data to the model 300 and obtains output data (e.g., images) from the model 300. The output data is passed to the output unit 230. In S203, the output unit 230 outputs the output data. Images for each time point are output, allowing them to be viewed as a video.

[0018] <Device with both learning and inference functions> Figure 5 shows an example configuration of an information processing device 400, which has both learning and inference functions.

[0019] As shown in Figure 5, the information processing device 400 comprises an input unit 410, an arithmetic unit 420, and an output unit 430. The arithmetic unit 420 holds the model 300. The arithmetic unit 420 has the functions of both the learning unit 120 and the inference unit 220 described above. That is, the arithmetic unit 420 learns the model 300 and performs inference processing using the learned model 300. The output unit 430 outputs an image which is the result of the inference.

[0020] In the following, we will first describe the prior art and its problems in more detail, and then describe the technology according to this embodiment. Note that the learning device 100 and inference device 200 described above may be used to execute the prior art, such as NeRF. Therefore, the following description of the operation of the prior art may be described as the operation of the learning device 100 and inference device 200.

[0021] (Regarding conventional technology) As one type of XR content, free-viewpoint video that can be viewed from any viewpoint and angle is attracting attention.

[0022] One technique for creating free-viewpoint video involves placing multiple cameras around a subject to capture images, generating 3D data, and then using that 3D data to enable viewing the video from any position and angle.

[0023] <About NeRF (Neural Radiance Fields)> In recent years, Neural Radiance Fields (NeRF) (Non-Patent Literature 1) have attracted particular attention. In NeRF, as shown in Figure 6, an image is generated from an arbitrary viewpoint and angle by training a DNN (Deep Neural Network) with image data obtained by photographing an object from multiple viewpoints. Because NeRF allows for the direct prediction of an image from an arbitrary viewpoint without generating 3D data, information loss is reduced and realistic quality can be achieved.

[0024] The operation of the learning device 100 and the inference device 200 when NeRF is executed by the learning device 100 and the inference device 200 will be explained with reference to Figures 7 and 8, which are excerpts from Non-Patent Document 1.

[0025] The learning device 100 receives images of an object taken from various angles as training data. The learning unit 120 uses the neural network model (i.e., function) F θ It has F θ This is a model (function) that takes three-dimensional coordinates (x, y, z) and line of sight direction (θ, φ) as input and outputs brightness (r, g, b) and density (σ).

[0026] The learning unit 120 is F θ Using the brightness and density obtained, volume rendering is used to calculate the pixel value C(r) corresponding to the camera ray r and generate an image. Figure 7 shows an image of volume rendering.

[0027] More specifically, C(r) can be calculated using equations (1) and (2) described in Non-Patent Document 1, but since it is difficult to calculate the integral exactly on a computer, in practice, the integral value is approximately calculated using the Riemann sum method, which involves random sampling in each interval, using equations (4) and (5) described in Non-Patent Document 1 (= equations shown in Figure 7). In the equations shown in Figure 7, i represents a sample, and δ ｉ c is the distance between adjacent samples. ｉ is luminance, σ ｉ This is density.

[0028] More specifically, in NeRF, hierarchical sampling is performed, where sampling is evenly done in the coarse model, aiming at areas where fine details should be grasped, and focusing on sampling those areas in the fine model.

[0029] As shown in FIG. 8, the learning unit 120 calculates the error (Loss) between the estimated image obtained by volume rendering based on the output from the model and the correct image given as learning data, and optimizes F so that the Loss is minimized. More specifically, as shown in the equation in FIG. 8, the loss function is the sum of the reconstruction errors of the coarse model and the fine model. θ The inference unit 220 of the inference device 200 includes the learned model F. When an arbitrary viewpoint (line-of-sight direction) is given to the inference unit 220, pixel values are calculated by volume rendering with the output from the model F in the same manner as in the learning operation, and an image of the object seen from that viewpoint is output.

[0030] Regarding <Dynamic NeRF>, research on Dynamic NeRF, which realizes the creation of free-viewpoint videos at any time in a dynamic scene by extending NeRF in the time direction, is also actively conducted. That is, Dynamic NeRF is a method of learning video data obtained by shooting an object from multiple viewpoints, generating an image from an arbitrary viewpoint at regular intervals, and connecting the generated images in the time direction to generate a video from an arbitrary viewpoint. θ FIG. 9 shows an image of Dynamic NeRF. In Dynamic NeRF, an object is video-shot from multiple viewpoints to generate a dynamic scene at an arbitrary viewpoint. Since Dynamic NeRF can generate a very realistic video compared to existing free-viewpoint video generation methods, it is expected to provide more immersive XR content and telepresence experiences through services that utilize this. θ

[0031]

[0032]

[0033] The main methods for representing dynamic scenes in Dynamic NeRF proposed so far can be broadly classified into three categories. The first is a time-aware method that adapts to dynamic scenes by inputting time as a fourth coordinate following spatial coordinates. In this method, rendering is performed using independent sampling points for each time point. One of the challenges is that the amount of memory required increases significantly because the representation space changes from 3D to 4D compared to methods for static scenes, and each method attempts to reduce the amount of memory used by performing efficient spatial representation and feature interpolation.

[0034] The second method involves performing volume rendering within a deformation space obtained through canonical mapping. Canonical mapping is a technique that defines a certain time as the canonical space and represents points in space at other times by the movement and change of points in the canonical space. Furthermore, volume rendering is performed on points on a straight line at the time defined as the canonical space, and on points on a curve connecting the destination points at other times. As a result, rendering at any given time refers to the space at the time defined as the canonical space, thus guaranteeing temporal consistency in the time changes for each point in space.

[0035] The third method is an extension of 3D Gaussian Splatting (3DGS) in the time domain. 3DGS initializes space using a coarse point cloud and represents the space using a 3D Gaussian obtained by assigning brightness, density, and a covariance matrix based on the assumption of a three-dimensional Gaussian distribution, expressed by spherical harmonics, to each point. Using a rendering method called rasterization, each 3D Gaussian is projected onto the image plane based on the camera's line of sight, and then the pixel value is calculated by calculating a weighted sum in depth order. This allows for faster rendering compared to volume rendering performed with NeRF.

[0036] In particular, K-Planes (Non-Patent Document 2), one of the first methods mentioned above, has served as a reference for many other methods.

[0037] In K-Planes, instead of directly predicting brightness and density in 3D space using coordinates and time, feature vectors are stored in a latent space obtained by compressing 4D space into six grid planes. Brightness and density are then predicted by inputting the Hadamard product of the feature vectors obtained by linearly interpolating from grids located near the input coordinates for each plane into a Multi-Layer Perceptron (MLP). Figure 10 (excerpt from Non-Patent Literature 2) shows the method for obtaining feature vectors.

[0038] Upon receiving input coordinates, the feature vectors of the four nearest neighbor points on a grid on a plane are obtained through linear interpolation. Furthermore, to ensure consistency across various scales, a scale parameter S is defined, allowing the final feature vector to be obtained through multi-scale linear interpolation. This linear interpolation is also performed on planes including the time axis, i.e., the xt, yt, and zt planes, making it possible to obtain feature vectors even for unknown time points.

[0039] (Regarding the issues) The following issues are common to both the first and second embodiments.

[0040] In creating free-viewpoint video in dynamic scenes, there is a demand to create video consisting of frames from any given time. If this technology is realized, for example, it will be possible to create video at a typical frame rate (e.g., 30 fps) using a model trained on low-frame-rate video (e.g., 5 fps). This will reduce the amount of data and conserve computing resources.

[0041] Furthermore, it becomes possible to create high-frame-rate video (e.g., 120fps) using a model trained on standard frame-rate video (e.g., 30fps). This enables the creation of high-quality video experiences and the provision of slow-motion video.

[0042] However, existing methods such as K-Planes mentioned above are not designed to perform inferences about unknown times (arbitrary times), and are trained on the premise of performing inferences about known times. Therefore, there is a problem in that the quality of the generated video deteriorates when inferences are performed about unknown times using existing methods. When we conducted an experiment using K-Planes to perform inferences about unknown times, we confirmed that the video quality deteriorated significantly.

[0043] (Outline of the technology according to this embodiment) In this embodiment, a mechanism is designed to predict deformation information at intermediate time points by referring to video frame interpolation (VFI), and by incorporating this mechanism into Dynamic NeRF, the above problems are solved, and Dynamic NeRF capable of high-precision inference for any given time point is realized.

[0044] In other words, in this embodiment, for example, the information processing device 400 learns video data obtained by photographing an object from multiple viewpoints, generates information corresponding to an image from an arbitrary viewpoint at a learned time, and then transforms the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time, thereby generating an image from an arbitrary viewpoint at an arbitrary time.

[0045] Video frame interpolation is the task of generating intermediate frames from two images (keyframes). In the learning phase, the system learns to predict the flow to the keyframes and reconstruct them by warping the predicted frames. Furthermore, it is possible to generate smooth video by recursively generating intermediate frames.

[0046] (Overview of video frame interpolation) In this embodiment, the RIFE-based technology disclosed in Non-Patent Document 4 is used as the video frame interpolation technology, so the overview of the RIFE-based video frame interpolation process will be explained here. Figure 11 is an excerpt (Fig. 2) from Non-Patent Document 4.

[0047] For the sake of explanation, we will refer to the entity that performs the interpolation process in RIFE as the "video interpolation model." The video interpolation model is a pair of consecutive RGB frames. ０ and I １ , and time t (0 ≤ t ≤ 1) are inputs, and the intermediate frame ^I at time t ｔ It generates and outputs the following. Note that in the text of the specification, "^I" is intended to mean that "^" is placed above "I".

[0048] The video interpolation model is I ０ , I １ By inputting the time t into IFNet, the intermediate flow F ｔ→０ F ｔ→１ , and estimate the fusion map M, and use the following formula to determine the intermediate frame ^I ｔ Generates.

[0049] In the above equations (1) and (2), ← W represents reverse warping, the symbol with a dot inside a circle represents the Hadamard product (element-wise product), and M (0 ≤ M ≤ 1) represents the fusion map. Note that Figure 11 (Fig. 2 of Non-Patent Literature 4) shows the correct image I. ＧＴ ｔ It has been shown that IFNet is trained by calculating loss by comparing the flow image obtained with the estimated flow image.

[0050] More specifically, the video interpolation model first uses IFNet to handle the intermediate flow between frames F ｔ→０ and F ｔ→１ The movement of each pixel is estimated. Next, the video interpolation model warps each frame in reverse based on the estimated intermediate flow, as shown in equation (2) above, to create intermediate frame ^I. ｔ←０ and ^I ｔ←１ To obtain.

[0051] Finally, the video interpolation model, as shown in equation (1) above, fuses the warped frames using the fusion map M to produce the final intermediate frame ^I. ｔ Generates.

[0052] Warping, in the context of an image (frame), means moving pixels from their original position to a new position, thereby transforming the frame. Fusion Map M is the frame after warping (^I ｔ←０ and ^I ｔ←１ This is weighting information used to determine how much information to incorporate. IFNet is a neural network for estimating intermediate flows.

[0053] (Configuration and operation of the learning unit 120 in the learning device 100) Figure 12 shows an example of the configuration of the learning unit 120 having a model 300 that incorporates video frame interpolation.

[0054] As shown in Figure 12, the learning unit 120 comprises a model 300 and a parameter update unit 121. The model 300 includes a dynamic NeRF unit 310, an intermediate flow prediction unit 320, a brightness / density prediction unit 330, and a rendering unit 340. The functions and operations of each part in the learning unit 120 will be explained with reference to Figure 13.

[0055] <Meaning of Symbols> The meaning of the symbols in Figure 13 is as follows:

[0056] C: Brightness (RGB) σ: Density (opacity) x: 3D coordinates d: Line of sight vector t: Time i: Key frame time I ｔ ｃａｍ : Camera cam frame image at time t F ｔ→ｉ : Transformation information from time t to keyframe time i. In the following explanation, two adjacent keyframe times will be represented by 0 and 1. In this case, any time x between two adjacent keyframe times is 0 < x < 1.

[0057] <Operation of each part and flow of operation> As a prerequisite for the operation shown in Figure 13, it is assumed that a dynamic NeRF unit 310 has been prepared, which has been generated (learned) from images of an object captured by multiple cameras. Furthermore, it is assumed that the pose, coordinates, camera parameters, and frame time of each camera are known, and that time synchronization is maintained between the multiple cameras. The frames of the images used for learning are called keyframes.

[0058] The dynamic NeRF unit 310 may use any method in Dynamic NeRF. For example, it may execute the NeRF processing described with reference to Figures 7 and 8 at each keyframe time. Alternatively, the dynamic NeRF unit 310 may be based on the k-planes technology described above.

[0059] Model 300 interpolates between keyframes at two adjacent time points to generate video from an arbitrary viewpoint. The parameter update unit 121 performs a loss calculation between the estimated image output from Model 300 and the ground truth image, and updates the parameters of Model 300 to minimize the loss.

[0060] In this embodiment, the parameters of the intermediate flow prediction unit 320 in model 300 are updated. Also, in this embodiment, it is assumed that the dynamic NeRF unit 310 is prepared in advance, but the dynamic NeRF unit 310 may be learned together with the intermediate flow prediction unit 320.

[0061] As shown in Figure 13, the dynamic NeRF unit 310 is a sparse-time dynamic NeRF, and x, d, t i The input is x, d, t i Brightness C corresponding to ｉ and density σ ｉ Outputs.

[0062] The intermediate flow estimator 320 predicts the deformation information (flow) at intermediate time points. The intermediate flow estimator 320 is a neural network similar to the IFNet mentioned above.

[0063] More specifically, the intermediate flow prediction unit 320 takes x, d, and t as inputs and calculates transformation information (intermediate flow) F from time t (0 < t < 1) in spatial coordinate x to keyframe times 0 and 1 respectively. ｔ→０ and F ｔ→１ And a weight M for superimposing bidirectional predictions. ｔ This predicts that bidirectional deformation information and weights will be predicted. F ｔ→０ and F ｔ→１These are each vector values.

[0064] The luminance / density prediction unit 330 (Point warping) receives "F" from the intermediate flow prediction unit 320. ｔ→０ F ｔ→１ M ｔ " and {C from the dynamic NeRF unit 310 ｉ , σ ｉ |i=0, 1} (specifically C ０ , σ ０ , C １ , σ １ ) is the input, and the image of the intermediate frame at time t is restored, such as "luminance ^C corresponding to x, d, t". ｔ and density ^σ ｔ " predicts."

[0065] Specifically, by replacing I with C in the aforementioned equations (1) and (2), the luminance ^C can be calculated. ｔ The density ^σ is calculated by replacing I with σ in the aforementioned equations (1) and (2). ｔ Calculate.

[0066] In Figure 13, "Color" corresponds to C, and "Opacity" corresponds to σ. ０ →CC １ , σ ０ →σ １ When the changes are visualized continuously, it becomes a color bar (a bar showing shades of gray) as shown in Figure 13, C ｔ and σ ｔ Determining this means identifying the correct position from this color bar.

[0067] Next, the rendering unit 340 (Volume rendering) calculates the luminance C corresponding to x, d, and t for various x and d values. ｔ and density ^σ ｔ Using this, volume rendering is performed on the light rays emitted from each pixel, similar to conventional NeRF, to generate an image at time t.

[0068] Next, the parameter update unit 121 calculates the error in pixel values ​​(RGB) between the generated image at time t and the ground truth image at time t as a loss, and updates the parameters of the intermediate flow prediction unit 320 to minimize the loss.

[0069] <Variations> In addition to the error in the pixel values ​​(RGB) used in the above loss calculation, the "F" calculated from the ground truth image at time t and keyframes 0 and 1 is also used. ｔ→０ F ｔ→１ M ｔ The value obtained by projecting "" onto a 2D image viewed from a certain viewpoint using volume rendering, and the value of "F" predicted by the intermediate flow prediction unit 320. ｔ→０ F ｔ→１ M ｔ The error between the corresponding pixel value of " may be used. In this case, the learning unit 120 selects the correct image at time t and keyframes 0 and 1 from "F ｔ→０ F ｔ→１ M ｔ It has a function to calculate "F" (this function may also be called the video frame interpolation function). Furthermore, when using the information processing device 400 in Figure 5, the calculation unit 420 has this function. In addition, "F" is calculated from the correct image at time t and keyframes 0 and 1. ｔ→０ F ｔ→１ M ｔ This can be called the transformed information of the correct answer.

[0070] Furthermore, it is possible to perform training using only keyframes, without using the ground truth image at time t.

[0071] Specifically, the intermediate flow prediction unit 320 uses F as transformation information from time 1 to time 0. １→０ The brightness / density prediction unit 330 predicts C ０ , σ ０ F １→０ From brightness ^C １ and density ^σ １ The rendering unit 340 predicts and renders image I at time 1. １ Generates.

[0072] Similarly, the intermediate flow prediction unit 320 uses F as transformation information from time 0 to time 1.０→１ is predicted, and the luminance / density prediction unit 330 predicts luminance ^C １ , σ １ , F ０→１ from, and predicts luminance ^C ０ and density ^σ ０ , and the rendering unit 340 generates an image I at time 0 ０ .

[0073] The parameter update unit 340 updates the parameters so that the error between the predicted image I １ and the correct image I １ , and the error between the predicted image I ０ and the correct image I ０ are minimized.

[0074] In the above method, it is not necessary to prepare the image at time t separately from the learning of the dynamic NeRF unit 310, and learning can be performed with less learning data.

[0075] (Operation of the inference unit 220) In the inference phase, the inference unit 220 in the inference device 200 uses the learned model 300 to input arbitrary "d, t" and outputs an image corresponding to "d, t". By moving t, it can be viewed as a video (image). Also, by changing the granularity of t, the quality (frame - per - second) of the video can be changed.

[0076] [Second Embodiment] Next, the second embodiment will be described. In the second embodiment, the internal configuration and learning method of the model 300 are different from those of the first embodiment. The device configuration and the overall operation of the device are the same in the first and second embodiments. Hereinafter, the description will focus on the differences from the first embodiment.

[0077] (Model Configuration) FIG. 14 shows the configuration of the model 300 in the second embodiment. The model 300 in the second embodiment shown in FIG. 14 has the intermediate flow prediction unit 320 replaced by a deformation information prediction unit 350 compared to the model 300 in the first embodiment shown in FIGS. 12 and 13. The configurations of the other parts are the same as those of the first embodiment.

[0078] The deformation information prediction unit 350 in the second implementation is a 3D flow predictor that can work in conjunction with the existing Dynamic NeRF (dynamic NeRF unit 310 in Figure 14). The deformation information prediction unit 350 may also be called BIFE3D (Bidirectional Intermediate Flow Estimation in 3D space).

[0079] The deformation information prediction unit 350 is a functional unit that predicts bidirectional 3D flow at intermediate time, and specifically, it predicts deformation information F from time t (0 < t < 1) to keyframe time in spatial coordinate p. ｔ－＞ｉ Predict (p, d, t).

[0080] (Operation during inference) An example of the operation of the model 300 when it is used for inference in the configuration shown in Figure 3 will be described. In particular, the operation of the deformation information prediction unit 350 will be described in detail. Note that the operation during learning also includes the operation described below.

[0081] As a prerequisite for operation, it is assumed that a Dynamic NeRF unit 310 (Dynamic NeRF model) has been acquired using video footage of an object captured by multiple cameras. Furthermore, it is assumed that the pose, coordinates, camera parameters, and frame time of each camera are known, and that time synchronization is maintained between the multiple cameras.

[0082] As described in the first embodiment, there are various methods for Dynamic NeRF, but the method adopted for the Dynamic NeRF unit 310 is not limited to a specific method. However, here we assume the use of the k-planes mentioned above. Hereafter, the neural network model corresponding to the deformation information prediction unit 350 will be referred to as "BIFE3D".

[0083] The problem in this embodiment is to perform three-dimensional frame interpolation at arbitrary time points between adjacent frames in Dynamic NeRF. The time points where adjacent frames have been learned in Dynamic NeRF are called keyframe times, and by performing the following processing from the beginning to the end of the time points, three-dimensional frame interpolation can be performed for all time points.

[0084] The following describes the image generation process at an arbitrary viewpoint d and an arbitrary time t (0 < t < 1). The model 300 infers an image at an arbitrary viewpoint d and an arbitrary time t using three-dimensional information obtained from the dynamic NeRF unit 310 at the keyframe time and deformation information predicted from BIFE3D. Note that the three-dimensional information obtained from the dynamic NeRF unit 310 at the keyframe time is the luminance and density at the spatial coordinates (x, y, z) at the viewpoint d at the keyframe time t.

[0085] In the inference executed by the model 300, the same process occurs for each pixel of the generated image. Therefore, here, the process for the inference of only one pixel will be described.

[0086] <Regarding the deformation of the rendering ray> In order to predict the information at time t using the three-dimensional information at the keyframe time, it is necessary to deform the rendering ray used in volume rendering. This is synonymous with the object having moved with time change.

[0087] Note that the rendering ray is a virtual line (ray) emitted from the viewpoint (camera) into the space when generating an image (2D) from three-dimensional information. Each rendering ray corresponds to one pixel of the image. In the present embodiment, BIFE3D is used to deform each sample point on the rendering ray according to time t. The specific processing content is as follows.

[0088] BIFE3D predicts the displacement vector (coordinate change amount) to the corresponding point at each keyframe time (e.g., time 0 and 1) corresponding to the point of interest (x, y, z) at time t.

[0089] That is, BIFE3D takes the three-dimensional coordinates (x, y, z) of the point of interest sampled on the rendering ray and time t as inputs, and the coordinate change amount (Δx ０ , Δy ０ , Δz ０ , Δx １ , Δy １ , Δz １) predicts. In other words, the processing of BIFE3D is "Δx ０ , Δy ０ , Δz ０ , Δx １ , Δy １ , Δz １ It can be written as "=BIFE3D(x,y,z,t)".

[0090] Here, (Δx ０ , Δy ０ , Δz ０ ) is the amount of coordinate change from the point of interest at time t to the corresponding point at time 0, and (Δx １ , Δy １ , Δz １ ) represents the change in coordinates from the point of interest at time t to the corresponding point at time 1.

[0091] For keyframe time 0, the change amount (Δx) predicted from BIFE3D ０ , Δy ０ , Δz ０ Adding ) to the coordinates of interest (x, y, z) creates a new coordinate system (x + Δx ０ , y + Δy ０ , z + Δz ０ ) is equal to the coordinates of the corresponding point of interest at time 0.

[0092] The deformation information prediction unit 350 obtains a deformed rendering ray by performing the above calculation for all sampling points on the rendering ray.

[0093] <Prediction of pixel values ​​in the image at time t> Next, using the information from the dynamic NeRF unit 310 at time 0, brightness and density are predicted using a conventional method. That is, the brightness / density prediction unit 330 is calculated as follows: "viewpoint d, time 0, coordinate (x + Δx ０ , y + Δy ０ , z + Δz ０ By inputting ")" to the dynamic NeRF unit 310, the dynamic NeRF unit 310 will receive "viewpoint d, time 0, coordinates (x + Δx ０ , y + Δy ０ , z + Δz ０The brightness and density corresponding to the ) are output, and the brightness / density prediction unit 330 acquires the brightness and density. By performing this process at all sampling points on the deformed rendering ray, the brightness and density on the deformed rendering ray can be obtained.

[0094] The brightness / density prediction unit 330 inputs the acquired brightness and density to the rendering unit 340. Alternatively, the rendering unit 340 may acquire brightness and density from the dynamic NeRF unit 310 without the brightness / density prediction unit 330. The rendering unit 340 predicts the pixel values ​​in the image at time t by performing volume rendering using the above brightness and density. In other words, it predicts the pixel values ​​in the image at time t from the keyframe time 0 information learned by the dynamic NeRF unit 310. The image at time t is generated by performing the above process with various rendering rays so that an image can be constructed.

[0095] <Regarding Keyframe Time 1> The same applies to Keyframe Time 1. That is, for Keyframe Time 1, the amount of change (Δx) predicted from BIFE3D １ , Δy １ , Δz １ Adding ) to the coordinates of interest (x, y, z) creates a new coordinate system (x + Δx １ , y + Δy １ , z + Δz １ ) is equal to the coordinates of the corresponding point of interest at time 1.

[0096] Viewpoint d, time 1, coordinates (x + Δx) １ , y + Δy １ , z + Δz １ By inputting ")" to the dynamic NeRF unit 310, the dynamic NeRF unit 310 will receive "viewpoint d, time 1, coordinates (x + Δx １ , y + Δy １ , z + Δz １ It outputs the brightness and density corresponding to ). By performing this calculation at all sampling points on the transformed rendering ray, the brightness and density on the transformed rendering ray can be obtained.

[0097] The rendering unit 340 predicts the pixel values ​​in the image at time t by performing volume rendering using the brightness and density described above. In other words, it predicts the pixel values ​​in the image at time t from the keyframe time 1 information that has been learned by the dynamic NeRF unit 310. The image at time t is generated by performing the above process with various rendering rays so that an image can be constructed.

[0098] As described above, Model 300 takes two images at viewpoint d and time t, one each from two directions at time 0 and 1 (I ｔ＜－０ , I ｔ＜－１ ) Generate.

[0099] The above processes can be summarized and written as follows.

[0100] I ｔ＜－０ =K-Planes(x+Δx ０ , y + Δy ０ , z + Δz ０ ,d,0) ・I ｔ＜－１ =K-Planes(x+Δx １ , y + Δy １ , z + Δz １ d, 1) Here, I ｔ＜－０ This is the image of time t predicted by deformation from time 0, I ｔ＜－１ This is the image at time t predicted by deformation from time 1.

[0101] <Image of Coordinate Change Amount> Figures 15 and 16 show an image of the coordinate change amount predicted from BIFE3D. Figure 15 shows a moving person, the line of sight vector d passing through it, and a sample point P at time t (e.g., 0.5). Point P is the point of interest on the rendering ray.

[0102] Figure 16 shows which point P in Figure 15 corresponds to at keyframe time (0 and 1). The coordinate change from point P to the corresponding point is F. ｔ－＞０ = (Δx ０ , Δy ０ , Δz ０ ,), F ｔ－＞１ = (Δx １ , Δy １ , Δz１ It can be expressed as ).

[0103] <Regarding processing during inference> During inference (when generating images), the model 300 (inference unit 220) uses the following as the image at viewpoint d and time t: ｔ＜－０ and I ｔ＜－１ You can output an image of the average (calculated by adding up the values ​​for each pixel and dividing by two), or you can output an image of the weighted sum.

[0104] Furthermore, as will be described later, the deformation information prediction unit 350 is I ｔ＜－０ and I ｔ＜－１とが Since it is learned to match, during inference, I ｔ＜－０ and I ｔ＜－１ You may generate and output one of the two images.

[0105] (Learning Operation) Next, the learning operation of the model 300 will be explained. Here, the model 300 is provided within the learning unit 120 shown in Figures 1 and 12, and the learning unit 120 performs the learning of the model 300. In the second embodiment, the model 300 has the configuration shown in Figure 14, and in Figure 12, the intermediate flow prediction unit 320 is replaced by the deformation information prediction unit 350.

[0106] Here, it is assumed that the dynamic NeRF unit 310 has already been trained, so during training, the parameters of the deformation information prediction unit 350, which is composed of a neural network, are updated and optimized. However, the dynamic NeRF unit 310 may also be included in the training target.

[0107] The following describes the specific processing performed by the learning unit 120. During learning, the model parameters are updated to minimize the loss. In this embodiment, multiple types of losses are used. The losses calculated by the parameter update unit 121 are as follows.

[0108] <rgb ｍｉｄ＿ｌｏｓｓ > First, RGB ｍｉｄ＿ｌｏｓｓ This will be explained. Model 300 operates in the same way as during the inference described above, predicting image I at time t from the information of viewpoint d and time 0. ｔ＜－０ And, image I at time t predicted from the information of viewpoint d and time 1.ｔ＜－１ Generates.

[0109] I is an RGB image predicted from both directions. ｔ＜－０ and I ｔ＜－１ They must be the same image. Therefore, the parameter update unit 121, ｔ＜－０ and I ｔ＜－１ By taking the Mean Square Error (MSE) for rgb, ｍｉｄ＿ｌｏｓｓ = MSE(I ｔ＜－０ , I ｔ＜－１ ) is calculated. In other words, I ｔ＜－０ and I ｔ＜－１ For each corresponding pixel, the difference is squared and the average is taken across all pixels to obtain RGB. ｍｉｄ＿ｌｏｓｓ Calculate.

[0110] <Processing using RIFE> The parameter update unit 121 performs loss calculations using information obtained from RIFE, so the processing using RIFE will be explained here. As mentioned above, RIFE is a video frame interpolation technology disclosed in Non-Patent Literature 4. Specifically, as shown below, I ０ , I １ With t as input, I ｔ , F ｔ－＞０ , F ｔ－＞１ It outputs the following. Note that in this embodiment, RIFE is used, but this is just one example. ０ , I １ With t as input, I ｔ , F ｔ－＞０ , F ｔ－＞１ Any video frame interpolation technique may be used to output the desired result.

[0111] I ｔ , F ｔ－＞０ , F ｔ－＞１ =RIFE(I ０ , I １ ,t) ・I ｔ : Interpolated image at viewpoint d and time t (image predicted by RIFE) ・F ｔ－＞０ Optical flow F from time t to time 0 ｔ－＞１ Optical flow I from time t to time 1 ０ Image I predicted from viewpoint d at time 0.１ : The image predicted from viewpoint d at time 1. Note that optical flow is a two-dimensional vector field that indicates where each pixel in one frame (e.g., time t) corresponds to in another frame (e.g., time 0).

[0112] <flow ｌｏｓｓ＿０ , flow ｌｏｓｓ＿１ > Next, flow ｌｏｓｓ＿０ , flow ｌｏｓｓ＿１ This will be explained. The parameter update unit 121 uses the optical flow prediction model RAFT (Recurrent All-Pairs Field Transforms for Optical Flow) to predict the image I at time t. ｔ＜－０ and the predicted image I at keyframe time 0 ０ Optical flow RAFT (I) between ｔ＜－０ , I ０ ) predicts the predicted image I at time t. ｔ＜－１ and the predicted image I at keyframe time 1 １ Optical flow RAFT (I) between ｔ＜－１ , I １ ) predicts.

[0113] RAFT is a model disclosed in Non-Patent Document 5, which takes two images as input and predicts bidirectional optical flow between them. It models a coarse-to-fine inference process using a Recurrent Neural Network (RNN), enabling high-precision, high-speed, and memory-efficient optical flow inference. Note that using RAFT is just one example; any technique that predicts optical flow between two input images may be used instead of RAFT.

[0114] Next, the parameter update unit 121 takes the MSE between the pseudo-ground truth optical flow predicted from RIFE and the optical flow predicted from RAFT. Through the above process, the flow is as follows: ｌｏｓｓ＿０ and flow ｌｏｓｓ＿１ Calculate.

[0115] ・flow ｌｏｓｓ＿０ = MSE(Fｔ－＞０ ,RAFT(I ｔ＜－０ , I ０ )) ・flow ｌｏｓｓ＿１ = MSE(F ｔ－＞１ ,RAFT(I ｔ＜－１ , I １ )) <rgb ｌｏｓｓ＿０ , rgb ｌｏｓｓ＿１ > Next, RGB ｌｏｓｓ＿０ , rgb ｌｏｓｓ＿１ This will be explained. The parameter update unit 121 updates the RGB image at time t, which was predicted at times 0 and 1. ｔ＜－０ and I ｔ＜－１ And, a pseudo-ground truth RGB image I predicted from RIFE. ｔ The MSE is taken between the two. This process results in the following rgb ｌｏｓｓ＿０ rgb ｌｏｓｓ＿１ Calculate.

[0116] RGB ｌｏｓｓ＿０ = MSE(I ｔ , I ｔ＜－０ ) ・ rgb ｌｏｓｓ＿１ = MSE(I ｔ , I ｔ＜－１ ) Note that RIFE is used in this embodiment, but this is just one example. Pseudo-ground truth RGB image I ｔ Any video frame interpolation technique that predicts the outcome may be used.

[0117] <Loss Calculation and Learning> The parameter update unit 121 ultimately calculates the loss using a linear sum as shown below, and updates and optimizes the model parameters of the neural network deformation information prediction unit 350 (BIFE3D) so that the loss is minimized.

[0118] Loss=(1-α) {(1-β)(flow ｌｏｓｓ＿０ +flow ｌｏｓｓ＿１ ) + β(rgb ｌｏｓｓ＿０ +rgb ｌｏｓｓ＿１ )} + α × rgb ｍｉｄ＿ｌｏｓｓ However, α and β are weight parameters determined by the sigmoid function. In the early stages of training, the inference images contain a lot of noise, so RGB is used after sufficient training has progressed. ｍｉｄ＿ｌｏｓｓThis is a method for applying it.

[0119] In other words, using a sigmoid function with input such as the progress of learning, α is close to 0 in the early stages of learning, increases as learning progresses, and becomes close to 1 when learning is sufficiently advanced. Note that when α = 1, rgb ｍｉｄ＿ｌｏｓｓ Only this is used for learning.

[0120] β is a parameter used to adjust for the difference in scale between the RGB image and the optical flow in the MSE. Note that in situations where the inference image does not contain much noise, even in the early stages of training, the loss is set to RGB. ｍｉｄ＿ｌｏｓｓ You may use only this for learning.

[0121] As explained above, in the training of the second embodiment, RGB images and optical flows at intermediate time points are generated using video frame interpolation technology (e.g., RIFE), and these are used as the training dataset to train the model. Furthermore, a loss function is used that ensures consistency through bidirectional interpolation.

[0122] <Image of Loss Calculation> Figure 17 shows the image of the loss calculation described above. In S1, rgb ｍｉｄ＿ｌｏｓｓ = MSE(I ｔ＜－０ , I ｔ＜－１ ) is calculated. Note that in Figure 17, rgb ｍｉｄ＿ｌｏｓｓ to "L Ｃｏｌｏｒ＿ｍｉｄ It is written as "[...]". Also, 0.5 is used as an example of t.

[0123] In S2, the complementary image I generated by RIFE. ｔ To obtain the following: Note that in Figure 17, the complementary image I ｔ to "L Ｃｏｌｏｒ It is written as ". S3 indicates rgb ｌｏｓｓ＿０ = MSE(I ｔ , I ｔ＜－０ ) and RGB ｌｏｓｓ＿１ = MSE(I ｔ , I ｔ＜－１ Calculate the result.

[0124] In S4, the pseudo-ground truth optical flow predicted from RIFE and the optical flow predicted from RAFT are used to flow ｌｏｓｓ＿０ = MSE(F ｔ－＞０ ,RAFT(I ｔ＜－０ , I ０ )) and flow ｌｏｓｓ＿１ = MSE(F ｔ－＞１ ,RAFT(I ｔ＜－１ , I １ )) is calculated. Note that Figure 17 is flow ｌｏｓｓ to "L Ｆｌｏｗ It is written as "".

[0125] (Example Hardware Configuration) Any of the devices described in this embodiment (learning device 100, inference device 200, information processing device 400) can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud. In the case of the learning device 100, the model 300 may not be included in the program. In the case of the inference device 200 (or information processing device 400), the program may include the model 300.

[0126] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0127] Figure 18 shows an example of the hardware configuration of the computer described above. The computer in Figure 18 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0128] The program that enables processing on the computer is provided on a recording medium 1001, such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0129] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the memory device 1003 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., based on a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0130] Furthermore, the functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0131] The system includes memory for storing computer programs, which include computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. The computer programs can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0132] (Effects of the Embodiments) As described above, the technology described in these embodiments (first and second embodiments) makes it possible to generate images viewed from any viewpoint with high accuracy using video data captured from multiple viewpoints.

[0133] <Notes> (Note 1) A free-viewpoint video generation method executed by an information processing device, wherein the device learns video data obtained by photographing an object from multiple viewpoints, generates information corresponding to an image from an arbitrary viewpoint at a learned time, and generates an image from an arbitrary viewpoint at an arbitrary time by deforming the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time. (Note 2) The free-viewpoint video generation method according to Note 1, wherein the information processing device predicts the brightness and density of the object when viewed from an arbitrary viewpoint at an arbitrary position coordinate and an arbitrary time using the mechanism, and calculates the pixel values ​​of an image from an arbitrary viewpoint at an arbitrary time using the brightness and density to generate the image. (Note 3) The free-viewpoint video generation method according to Note 1, wherein the information processing device generates an image from an arbitrary viewpoint at an arbitrary time in between image data obtained by photographing an object from multiple viewpoints at a certain time and image data obtained by photographing an object from multiple viewpoints at a later time. (Note 4) The free-viewpoint video generation method according to Note 1, wherein the information processing device predicts deformation information of brightness and density from an arbitrary time in spatial coordinates viewed from an arbitrary viewpoint to a first time before that time and a second time after that time, respectively, using the mechanism, and uses the deformation information to predict the brightness and density at an arbitrary time in spatial coordinates, thereby generating an image from an arbitrary viewpoint at an arbitrary time. (Note 5) A free-viewpoint video generation method executed by an information processing device equipped with a dynamic NeRF unit that has been trained at keyframe times, wherein the amount of coordinate change from the point of interest to the corresponding point at the keyframe time is predicted from the three-dimensional coordinates of the point of interest on the rendering ray and a specific time, the three-dimensional coordinates of the corresponding point are calculated using the amount of coordinate change, the brightness and density output from the dynamic NeRF unit are obtained by inputting the viewpoint, the three-dimensional coordinates of the corresponding point and the keyframe time to the dynamic NeRF unit, and an image at the specific time is generated by performing volume rendering using the brightness and density.(Note 6) An information processing device comprising a calculation unit that learns video data obtained by photographing an object from multiple viewpoints, generates information corresponding to an image from an arbitrary viewpoint at a learned time, and transforms the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time, thereby generating an image from an arbitrary viewpoint at an arbitrary time. (Note 7) The information processing device according to Note 6, wherein the calculation unit learns the mechanism using correct deformation information obtained using a video frame interpolation function. (Note 8) An information processing device comprising a dynamic NeRF unit that has been learned at keyframe times, a deformation information prediction unit that predicts the amount of coordinate change from the point of interest on the rendering ray to the corresponding point at the keyframe time from the three-dimensional coordinates of the point of interest and a specific time, and calculates the three-dimensional coordinates of the corresponding point using the amount of coordinate change, and a rendering unit that inputs the viewpoint, the three-dimensional coordinates of the corresponding point and the keyframe time to the dynamic NeRF unit, obtains the brightness and density output from the dynamic NeRF unit, and generates an image at the specific time by performing volume rendering using the brightness and density. (Note 9) A learning device for learning a model for generating an image of an object, wherein the model predicts deformation information from the time between two adjacent keyframe times in the image to each of the two keyframe times, uses the deformation information to predict the brightness and density of the spatial coordinates of the object as seen from a certain viewpoint at the time, uses the brightness and density to generate an image of the object as seen from the viewpoint at the time, and a learning unit updates the parameters of the model based on the error between the image generated by the model and the correct image at the time.(Note 10) A learning device for learning a model for generating an image of an object, the model comprising: a deformation information prediction unit that predicts a first amount of coordinate change from the point of interest to a first corresponding point at the first keyframe time and a second amount of coordinate change from the point of interest to a second corresponding point at the second keyframe time, based on the three-dimensional coordinates of a point of interest on a rendering ray and a specific time between adjacent first keyframe times and second keyframe times, and calculates the three-dimensional coordinates of the first corresponding point and the second corresponding point using the first and second coordinate change amounts; a rendering unit that generates a first image at the specific time based on the three-dimensional coordinates of the first corresponding point and the first keyframe time, and generates a second image at the specific time based on the three-dimensional coordinates of the second corresponding point and the second keyframe time, and a learning unit that learns the deformation information prediction unit so that the first image and the second image match. (Note 11) A learning device for learning a model for generating images of an object, the model comprising: a dynamic NeRF unit which has been learned at adjacent first keyframe times and second keyframe times; a deformation information prediction unit which predicts a first coordinate change amount from the point of interest to a first corresponding point at the first keyframe time and a second coordinate change amount from the point of interest to a second corresponding point at the second keyframe time, based on the three-dimensional coordinates of a point of interest on a rendering ray and a specific time between the first keyframe time and the second keyframe time, and calculates the three-dimensional coordinates of the first and second corresponding points using the first and second coordinate change amounts; a rendering unit which generates a first image at the specific time using the brightness and density obtained by inputting the three-dimensional coordinates of the first corresponding point and the first keyframe time to the dynamic NeRF unit, and generates a second image at the specific time using the brightness and density obtained by inputting the three-dimensional coordinates of the second corresponding point and the second keyframe time to the dynamic NeRF unit, A learning device comprising a learning unit that performs learning of the deformation information prediction unit so that the first image and the second image match.(Note 12) The learning device according to Note 10, wherein the learning unit performs learning of the deformation information prediction unit using the following: loss based on the difference between the first image and the second image; optical flow between the first image and the image at the first keyframe time; loss based on the difference between the optical flow between the specific time and the first keyframe time obtained by video frame interpolation processing; optical flow between the second image and the image at the second keyframe time; loss based on the difference between the optical flow between the specific time and the second keyframe time obtained by video frame interpolation processing; loss between the image at the specific time and the first image obtained by video frame interpolation processing; and loss between the image at the specific time and the second image obtained by video frame interpolation processing. (Note 13) A non-temporary storage medium storing a program that causes a computer to function as a calculation unit that generates an image from an arbitrary viewpoint at an arbitrary time by training it with video data obtained by photographing an object from multiple viewpoints, generating information corresponding to an image from an arbitrary viewpoint at a learned time, and transforming the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time. (Note 14) A program that causes a computer to function as a learning device that learns a model for generating images of an object, wherein the model predicts deformation information from the time between two adjacent keyframe times in the video to each of the two keyframe times, uses the deformation information to predict the brightness and density in the spatial coordinates of the object as seen from a certain viewpoint at the given time, and uses the brightness and density to generate an image of the object as seen from the given viewpoint at the given time, and the program is stored in a non-temporary storage medium storing a program that causes the computer to function as a learning unit that updates the parameters of the model based on the error between the image generated by the model and the correct image at the given time.(Note 15) A non-temporary storage medium storing a program that causes a computer to function as a rendering unit that generates an image at a specific time by inputting the viewpoint, the three-dimensional coordinates of the corresponding point and the keyframe time into the dynamic NeRF unit, obtaining the brightness and density output from the dynamic NeRF unit, and performing volume rendering using the brightness and density.

[0134] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0135] This patent application claims priority based on PCT / JP2024 / 41842, filed on November 26, 2024, and the entire contents of PCT / JP2024 / 41842 are incorporated herein by reference.

[0136] 100 Learning device 110 Input unit 120 Learning unit 121 Parameter update unit 130 Output unit 200 Inference device 210 Input unit 220 Inference unit 230 Output unit 300 Model 310 Dynamic NeRF unit 320 Intermediate flow prediction unit 330 Brightness / density prediction unit 340 Rendering unit 350 Deformation information prediction unit 400 Information processing device 410 Input unit 420 Calculation unit 430 Output unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A free-viewpoint video generation method executed by an information processing device, wherein the device learns video data obtained by photographing an object from multiple viewpoints, generates information corresponding to an image from an arbitrary viewpoint at a learned time, and generates an image from an arbitrary viewpoint at an arbitrary time by deforming the generated information in the time direction using a mechanism that predicts deformation information at an arbitrary time.

2. The free-viewpoint video generation method according to claim 1, wherein the information processing device predicts the brightness and density of the object as viewed from an arbitrary viewpoint at arbitrary position coordinates and arbitrary time using the mechanism, and calculates the pixel values ​​of an image from an arbitrary viewpoint at arbitrary time using the brightness and density to generate the image.

3. The free-viewpoint video generation method according to claim 1, wherein the information processing device generates an image from any viewpoint at any time in between, using image data obtained from multiple viewpoints captured at a certain time and image data obtained from multiple viewpoints captured at a later time.

4. The free-viewpoint video generation method according to claim 1, wherein the information processing device predicts, using the mechanism, deformation information of brightness and density from an arbitrary time in spatial coordinates viewed from an arbitrary viewpoint to a first time before that time and a second time after that time, and uses the deformation information to predict the brightness and density at an arbitrary time in spatial coordinates, thereby generating an image from an arbitrary viewpoint at an arbitrary time.

5. A free-viewpoint video generation method executed by an information processing device equipped with a dynamic NeRF unit that has been trained at keyframe times, comprising: predicting the amount of coordinate change from a point of interest on a rendering ray to a corresponding point at the keyframe time based on the three-dimensional coordinates of the point of interest and a specific time; calculating the three-dimensional coordinates of the corresponding point using the amount of coordinate change; obtaining brightness and density output from the dynamic NeRF unit by inputting the viewpoint, the three-dimensional coordinates of the corresponding point, and the keyframe time to the dynamic NeRF unit; and generating an image at the specific time by performing volume rendering using the brightness and density.

6. An information processing device comprising a calculation unit that generates an image from any viewpoint at any given time by training it with video data obtained by photographing an object from multiple viewpoints, generating information corresponding to an image from any viewpoint at a learned time, and then transforming the generated information in the time direction using a mechanism that predicts deformation information at any given time.

7. The information processing device according to claim 6, wherein the calculation unit learns the mechanism using correct deformation information obtained using the video frame interpolation function.

8. An information processing device comprising: a dynamic NeRF unit that has been trained at keyframe time; a deformation information prediction unit that predicts the amount of coordinate change from the point of interest on the rendering ray to the corresponding point at the keyframe time based on the three-dimensional coordinates of the point of interest and a specific time, and calculates the three-dimensional coordinates of the corresponding point using the amount of coordinate change; and a rendering unit that inputs the viewpoint, the three-dimensional coordinates of the corresponding point and the keyframe time to the dynamic NeRF unit, obtains the brightness and density output from the dynamic NeRF unit, and generates an image at the specific time by performing volume rendering using the brightness and density.

9. A learning device for learning a model that generates an image of an object, the learning device comprising: the model predicts deformation information from the time between two adjacent keyframe times in the image to each of the two keyframe times; uses the deformation information to predict the brightness and density of the spatial coordinates of the object as seen from a certain viewpoint at the time; uses the brightness and density to generate an image of the object as seen from the viewpoint at the time; and a learning unit that updates the parameters of the model based on the error between the image generated by the model and the ground truth image at the time.

10. A learning device for learning a model for generating an image of an object, the model comprising: a deformation information prediction unit that predicts a first coordinate change amount from the point of interest to a first corresponding point at the first keyframe time and a second coordinate change amount from the point of interest to a second corresponding point at the second keyframe time, based on the three-dimensional coordinates of a point of interest on a rendering ray and a specific time between adjacent first keyframe times and second keyframe times, and calculates the three-dimensional coordinates of the first and second corresponding points using the first and second coordinate change amounts; a rendering unit that generates a first image at the specific time based on the three-dimensional coordinates of the first corresponding point and the first keyframe time, and generates a second image at the specific time based on the three-dimensional coordinates of the second corresponding point and the second keyframe time, and a learning unit that learns the deformation information prediction unit so that the first image and the second image match.

11. A learning device for learning a model for generating images of an object, the model comprising: a dynamic NeRF unit that has been learned at adjacent first keyframe times and second keyframe times; a deformation information prediction unit that predicts a first coordinate change amount from the point of interest to a first corresponding point at the first keyframe time and a second coordinate change amount from the point of interest to a second corresponding point at the second keyframe time, based on the three-dimensional coordinates of a point of interest on a rendering ray and a specific time between the first keyframe time and the second keyframe time, and calculates the three-dimensional coordinates of the first and second corresponding points using the first and second coordinate change amounts; and a rendering unit that generates a first image at the specific time using the brightness and density obtained by inputting the three-dimensional coordinates of the first corresponding point and the first keyframe time to the dynamic NeRF unit, and generates a second image at the specific time using the brightness and density obtained by inputting the three-dimensional coordinates of the second corresponding point and the second keyframe time to the dynamic NeRF unit. A learning device comprising a learning unit that performs learning of the deformation information prediction unit so that the first image and the second image match.

12. The learning device according to claim 10, wherein the learning unit performs learning of the deformation information prediction unit using the following: loss based on the difference between the first image and the second image; optical flow between the first image and the image at the first keyframe time; loss based on the difference between the optical flow between the specific time and the first keyframe time obtained by video frame interpolation processing; optical flow between the second image and the image at the second keyframe time; loss based on the difference between the optical flow between the specific time and the second keyframe time obtained by video frame interpolation processing; loss between the image at the specific time and the first image obtained by video frame interpolation processing; and loss between the image at the specific time and the second image obtained by video frame interpolation processing.

13. A program that causes a computer to function as a computation unit that generates an image from any viewpoint at any given time by training it with video data obtained by photographing an object from multiple viewpoints, generating information corresponding to an image from any viewpoint at a learned time, and then transforming the generated information in the time direction using a mechanism that predicts deformation information at any given time.

14. A program for causing a computer to function as a learning device for learning a model that generates images of an object, wherein the model predicts deformation information from the time between two adjacent keyframe times in the image to each of the two keyframe times, uses the deformation information to predict the brightness and density of the spatial coordinates of the object as seen from a certain viewpoint at the time, and uses the brightness and density to generate an image of the object as seen from the viewpoint at the time, and the program for causing the computer to function as a learning unit that updates the parameters of the model based on the error between the image generated by the model and the ground truth image at the time.

15. A program to cause a computer to function as a dynamic NeRF unit that has been trained at keyframe times, a deformation information prediction unit that predicts the amount of coordinate change from a point of interest on a rendering ray to a corresponding point at the keyframe time based on the three-dimensional coordinates of the point of interest and a specific time, and calculates the three-dimensional coordinates of the corresponding point using the amount of coordinate change, and a rendering unit that generates an image at the specific time by inputting the viewpoint, the three-dimensional coordinates of the corresponding point and the keyframe time to the dynamic NeRF unit, obtaining the brightness and density output from the dynamic NeRF unit, and performing volume rendering using the brightness and density.

Images