Data processing device, method, and program

Abstract

A data processing device according to one embodiment of the present invention comprises a creation unit that, on the basis of a parameter indicating a condition at the time of capturing a two-dimensional image and text indicating each of a plurality of types of patterns of an element to be reproduced as a three-dimensional space in the two-dimensional image, creates a mask image set including a binary mask image corresponding to each of the plurality of types of patterns and a binary mask image indicating a region not included in a region indicated by the binary mask image; a learning unit that inputs the two-dimensional image and the parameter to a model that reproduces the three-dimensional space by Gaussian distribution, obtains a rendering image corresponding to a designated pattern, obtains an image that is an element product of the two-dimensional image and the mask image corresponding to the designated pattern, and performs learning of the model on the basis of a loss between the image and the rendering image; and an image processing unit that inputs a new two-dimensional image to the learned model to obtain a two-dimensional image corresponding to the designated pattern .

Description

Data processing apparatus, method and program

[0001] Embodiments of the present invention relate to a data processing apparatus, method and program.

[0002] In recent years, the 3DGS (3D Gaussian Splatting) method (see, for example, Non-Patent Document 1) has attracted attention as one of the techniques for reconstructing a three-dimensional space from captured video. 3DGS expresses a three-dimensional space by a large number of Gaussian distributions, and renders the Gaussian distributions in this three-dimensional space by specifying camera parameters, which are conditions at the time of imaging such as tilt and position, thereby generating a two-dimensional image from an arbitrary viewpoint to realize reconstruction of the three-dimensional space. In addition, a dynamic 3DGS method (see, for example, Non-Patent Document 2) for reconstructing a dynamic three-dimensional space by adding a time parameter to the Gaussian distribution has been proposed.

[0003] In application to actual services, it is difficult to generate an ideal three-dimensional space in which only objects desired by a user are reconstructed. The reason for this is that, as a problem of 3DGS, when there are objects that do not need to be reconstructed in the captured video, 3DGS reconstructs the three-dimensional space based on the captured video, so the unnecessary objects are included in the reconstruction result.

[0004] To address such issues, there have been proposed a method that uses inpainting technology to interpolate unnecessary object regions so that they have a natural appearance (see, for example, Non-Patent Document 3), and a method that edits a 3DGS model after learning by using 2D foundation models such as SAM (Segment Anything Model) and CLIP (Contrastive Language-Image Pre-Training) and adding feature quantities of the 2D foundation model to parameters of the Gaussian distribution of the 3DGS model (see, for example, Non-Patent Document 4).

[0005] Bernhard Kerbl, et, al, “3D Gaussian Splatting for Real-Time Radiance Field Rendering,” ACM Transactions on Graphics (TOG), 42, 4 (2023), 1-14Zhan Li, et, al, “Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis,” In Conference on Computer Vision and Pattern Recognition (CVPR), (2024)Zhiheng Liu, et, al, “InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior,” arxiv:2404.11613Shijie Zhou, et, al, “Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields,” arxiv:2312.0203

[0006] When using the Inpainting technique described above, unnecessary objects are removed and interpolated images are generated to train the 3DGS model. Therefore, it is not possible to control the 3DGS model after training by temporarily switching the display and hiding of certain objects, or by partially switching the display and hiding of certain objects.

[0007] While the aforementioned method of utilizing a 2D base model allows for control of the 3DGS model, such as switching the display and hiding of objects after training, it requires rendering not only the brightness and transparency of the Gaussian distribution but also the features of the 2D base model, as described above, according to the camera parameters. This compromises the real-time capabilities of the 3DGS model.

[0008] This invention was made in view of the above circumstances, and its purpose is to provide a data processing device, method, and program that can appropriately reconstruct a three-dimensional space.

[0009] A data processing device according to one aspect of the present invention includes: a creation unit that creates a set of mask images including a binary mask image corresponding to each of the multiple types of patterns and a binary mask image indicating a region not included in the region indicated by the binary mask image, based on parameters indicating the conditions at which a two-dimensional image to be processed was captured and text information indicating each of the multiple types of patterns of the element to be reproduced as a three-dimensional space in the two-dimensional image; a learning unit that inputs the two-dimensional image to be processed and parameters indicating the conditions at which the two-dimensional image was captured into a model that reproduces a three-dimensional space using a Gaussian distribution, obtains a rendering image from the model corresponding to a specified pattern among the multiple types of patterns, obtains a two-dimensional image which is the element product of the two-dimensional image to be processed and the mask image corresponding to the specified pattern from the set of mask images, and learns the model based on the loss between the two-dimensional image and the rendering image; and an image processing unit that inputs a new two-dimensional image to be processed into the learned model to obtain a two-dimensional image corresponding to the specified pattern among the multiple types of patterns.

[0010] A data processing method according to one aspect of the present invention is a method performed by a data processing device, comprising: a creation unit of the data processing device creating a set of mask images including a binary mask image corresponding to each of the multiple types of patterns and a binary mask image indicating a region not included in the region indicated by the binary mask image, based on parameters indicating the conditions at the time of capturing a two-dimensional image to be processed and text information indicating each of the multiple types of patterns of elements that are to be reproduced as a three-dimensional space in the two-dimensional image; a learning unit of the data processing device inputting the two-dimensional image to be processed and parameters indicating the conditions at the time of capturing the two-dimensional image to a model that reproduces a three-dimensional space using a Gaussian distribution by inputting the two-dimensional image as input; obtaining a rendering image from the model corresponding to a specified pattern among the multiple types of patterns, obtaining a two-dimensional image which is the element product of the two-dimensional image to be processed and the mask image corresponding to the specified pattern from the set of mask images, and learning the model based on the loss between the two-dimensional image and the rendering image; and an image processing unit of the data processing device inputting a new two-dimensional image to be processed to the learned model to obtain a two-dimensional image corresponding to the specified pattern among the multiple types of patterns.

[0011] According to the present invention, a three-dimensional space can be appropriately reconstructed.

[0012] Figure 1 is a diagram showing an example of application of the data processing device according to the first embodiment of the present invention. Figure 2 is a flowchart showing an example of the procedure for processing operations by the data processing device according to the first embodiment of the present invention. Figure 3 is a diagram illustrating an example of the creation of guide information by the data processing device according to the first embodiment of the present invention. Figure 4 is a diagram illustrating an example of model learning by the data processing device according to the first embodiment of the present invention. Figure 5 is a diagram showing an example of application of the data processing device according to the second embodiment of the present invention. Figure 6 is a flowchart showing an example of the procedure for processing operations by the data processing device according to the second embodiment of the present invention. Figure 7 is a diagram illustrating an example of text feature creation and model learning by the data processing device according to the second embodiment of the present invention. Figure 8 is a block diagram showing an example of the hardware configuration of a data processing device according to one embodiment of the present invention.

[0013] Embodiments relating to this invention will be described below. In this embodiment, taking advantage of the fact that the user has a three-dimensional spatial image that they wish to reconstruct, text describing the elements that the user wishes to reconstruct as three-dimensional space is added to the image taken by the user. In this case, text information indicating each of several types of patterns of the elements that are to be reproduced as three-dimensional space is added.

[0014] In this embodiment, a binary mask image generated using text is used as guide information to train a 3DGS model. In this embodiment, by associating text with each Gaussian distribution of the 3DGS model, the 3DGS model is controlled after training by displaying and hiding objects corresponding to the text, as well as adjusting parameters.

[0015] In this embodiment, by applying a configuration that links text with the Gaussian distribution of the 3DGS model, the 3DGS model can be controlled even after training. Furthermore, in this embodiment, a configuration can be applied in which the features of the 2D base model are not assigned to the parameters of the Gaussian distribution of the 3DGS model, thereby maintaining the real-time capabilities of the 3DGS model.

[0016] (First Embodiment) Figure 1 is a diagram showing an example of the application of a data processing device according to the first embodiment of the present invention. As shown in Figure 1, the data processing device 100 according to the first embodiment of the present invention comprises a dataset input unit 10, a guide information creation unit 20, a model learning unit 30, and a model control unit (image processing unit) 40.

[0017] Figure 2 is a flowchart showing an example of the processing procedure by a data processing device according to the first embodiment of the present invention. The dataset input unit 10 acquires a 2D image to be processed, captured by the user, and camera parameters corresponding to the 2D image, as dataset input, and passes these acquired 2D image and camera parameters, as well as initial point cloud data, to the model learning unit 30 (S10).

[0018] Here, the dataset input unit 10 acquires a 2D image, rather than a 2D image and camera parameters. The camera parameters corresponding to this 2D image may be derived using an existing 3D reconstruction tool such as COLMAP or Metashape (registered trademark) that employs SfM (Structure from Motion) and MVS (Multi-view Stereo) technology. Alternatively, the dataset input unit 10 may acquire a 2D video instead of a 2D image, and use images extracted from this video at frame intervals specified by the user.

[0019] When reconstructing the dynamic 3DGS model, the dataset input unit 10 derives time information based on the frame number and order of the 2D images of the 2D video, and assigns this to the camera parameters. The initial point cloud data used above may consist of feature point data created by COLMAP or Metashape, or data in which random point clouds are placed in 3D space.

[0020] Figure 3 illustrates an example of guide information creation by a data processing device according to the first embodiment of the present invention. The guide information creation unit 20 acquires text information input from the user, acquires a 2D image from the dataset input unit 10, creates a text set which is a set of text corresponding to the 2D image, and a mask image set which is a set of mask images corresponding to the 2D image, as guide information, and passes it to the model learning unit 30 (S20).

[0021] The text set is represented as shown in (1) below, and indicates the object that the user wishes to control in a 3D space reconstructed from video captured by the user. Here, n represents the number of texts.

[0022]

[0023] The guide information creation unit 20 generates a 2D image, such as the input image I_ shown in Figure 3(a). i and the text T_ corresponding to the input image j Regarding this, by using a segmentation model that takes text as input, such as Grounded SAM, a mask image M_(i,j) corresponding to each text is created as shown in (2) below. Here, Seg(•) represents the segmentation model. M_(i,j)=Seg(I_i,T_j) …(2) In addition, the guide information creation unit 20 may independently create a mask image for each of the multiple input images, and when training a dynamic 3DGS model, a tracking method may be used to create the mask image.

[0024] The guide information creation unit 20 creates a mask image in the text set T of the input image I_i, and then, as shown in (3) below, creates a mask image M_(i,0) that indicates the region not included in any of the n mask images, and adds the text T_0 corresponding to this image to the text set. The content of text T_0 is not particularly limited as long as it does not conflict with the content of the other text sets, and may be expressed as, for example, "base" or "background".

[0025]

[0026] Therefore, if the number of input images is N, the guide information creation unit 20 creates a set of mask images M, consisting of N × (n+1) images, corresponding to each input image and each set of text, as shown in Figure 3(b), as described in (4) below.

[0027]

[0028] The content of each text T_0, T_1, and T_2 that make up the text set corresponding to the mask image set M shown in Figure 3(b) is as follows: T_0: base T_1: track T_2: the track on the left and a person riding a bicycle

[0029] Furthermore, the guide information creation unit 20 may assign to the element T_j of the text set T a text set shown as shown in (5) below, in which the object indicated by text T_j is divided and shown in a hierarchical structure. Here, n_j indicates the number of texts in the text set shown in (5). Similarly, the guide information creation unit 20 may add text sets to text T_j1 in a hierarchical structure. As a result, a mask image set in a hierarchical structure, as shown in Figure 3(c), can be created.

[0030]

[0031] The content of each text T_0, T_1, T_11, T_2, T_21, and T_22 that make up the text set corresponding to the mask image set M shown in Figure 3(c) is as follows: T_0: base T_1: track T_11: left track T_2: person riding a bicycle T_21: person T_22: bicycle

[0032] FIG. 4 is a diagram for explaining an example of model learning performed by a data processing apparatus according to a first embodiment of the present invention. A model learning unit (30) acquires a two-dimensional image to be processed, camera parameters, and initial point cloud data from a dataset input unit (10), acquires a text set and a mask image set from a guide information generation unit (20), and learns a 3DGS model using the obtained acquisition results (S30). The model learning unit (30) passes the learned 3DGS model and the text set to a model control unit (40), and outputs the learned 3DGS model and the text set as a file.

[0033] Similarly to the normal learning of a 3DGS model, the model learning unit 30 uses the initial point cloud data to initialize the Gaussian distribution, and initializes position, color, transparency, scale, and rotation information. Next, the model learning unit 30 assigns a coefficient vector representing a correspondence relationship with text to each Gaussian distribution of the 3DGS model, and performs initialization by setting "τ_j=1". The coefficient vector is expressed by the following formula (6).

[0034]

[0035] Next, for the number of times specified by the user, similarly to the normal learning of a 3DGS model, the model learning unit 30 renders a viewpoint image corresponding to the camera parameters, which is expressed by the following formula (7), as shown in the following formula (8).

[0036]

[0037] The model learning unit 30 calculates a loss between the viewpoint image and the two-dimensional image I_i to be processed according to the following formula (9), and performs learning of the 3DGS model based on the loss.

[0038]

[0039] Here, C represents the color of one pixel constituting the viewpoint image, the following formula (10) represents a set of sorted Gaussian distributions overlapping a given pixel, c_g represents color information of the Gaussian distribution, and α_g represents transparency.

[0040]

[0041] Then, after normal training of the 3DGS model, the model learning unit 30 trains the 3DGS model using text and mask images. First, the model learning unit 30 creates a text indicator represented by the following formula (11), which indicates which text in the text set is valid and the image that the user desires to display as a viewpoint image.

[0042]

[0043] When the j'-th text in the text set is valid, that is, when one pattern is specified among the plurality of types of patterns indicated by each text in the text set, the text indicator is represented by the following formula (12), and the model learning unit 30 performs rendering processing according to the following formula (13).

[0044]

[0045] The model learning unit 30 obtains the loss according to the following formula (16) using the viewpoint image represented by the following formula (14) obtained by this rendering processing, that is, the rendered image corresponding to the above-mentioned valid text, the two-dimensional image Ii to be processed, and the mask image represented by the following formula (15), and trains the 3DGS model based on this loss. This loss corresponds to the loss between the two-dimensional image, which is the element-wise product of the two-dimensional image to be processed and the mask image corresponding to the specified pattern in the mask image set, and the two-dimensional image Ii to be processed.

[0046]

[0047] Although the description is given here under the condition that the number of valid texts is one, the 3DGS model may also be trained with the elements of a plurality of text indicators set to 1. When a plurality of texts are valid, the model learning unit 30 obtains the masked image Mi by taking the logical OR as shown in the following formula (17), and trains the 3DGS model by obtaining the loss between the viewpoint image and the Hadamard product, which is the element-wise product of this mask image and the two-dimensional image Ii to be processed.

[0048]

[0049] The model control unit 40 acquires the 3DGS model learned by the model learning unit 30, or a 3DGS model learned from a separately created 3DGS model file, and a text set, respectively. It then outputs the result of the rendering process when given camera parameters for an arbitrary viewpoint, and displays it on a display device (not shown) to control the 3DGS model, such as switching between partially displaying or hiding parts of it (S40).

[0050] The model control unit 40 has a text indicator that specifies whether to display or hide the text set, as shown in (11) above. The model control unit 40 provides the 3DGS model with camera parameters for an arbitrary viewpoint and performs rendering processing similar to that of the model learning unit 30, thereby displaying images corresponding only to the enabled text and corresponding objects as a new viewpoint image.

[0051] Here, enabling or disabling the text using the text indicator shown in (11) above can be specified by the user through command operations, button operations, or click operations on the GUI (Graphical User Interface).

[0052] Furthermore, the model control unit 40 can control the 3DGS model in ways other than displaying or hiding it, by adding the text parameters shown in (18) below to the 3DGS model in addition to the text indicator, and by specifying, for example, the resolution as an element of the parameter vector shown in (19) below.

[0053]

[0054] (Second Embodiment) Next, a second embodiment will be described. Detailed explanations of parts of this embodiment that overlap with the parts described in the first embodiment will be omitted. Figure 5 is a diagram showing an example of application of the data processing device according to the second embodiment of the present invention. As shown in Figure 5, the data processing device 100a according to the first embodiment of the present invention includes a data set input unit 10, a guide information creation unit 20, a model learning unit 30, and a model control unit 40 described in the first embodiment, in addition to a text feature generation unit 50.

[0055] Figure 6 is a flowchart illustrating an example of the processing procedure by a data processing device according to a second embodiment of the present invention. Figure 7 is a diagram illustrating an example of text feature creation and model learning by a data processing device according to a second embodiment of the present invention. In the second embodiment, a dataset is input by the dataset input unit 10 (S110), similar to S10 described in the first embodiment; guide information is created by the guide information creation unit 20 (S120), similar to S20 described in the first embodiment; and model learning is performed by the model learning unit 30 (S120), similar to S30 described in the first embodiment.

[0056] Then, the text feature generation unit 50 performs the text feature generation process described later (S140), and the model control unit 40 performs model control in the same way as described in S40 in the first embodiment (S150).

[0057] The text feature generation process will now be explained. The text feature generation unit 50 receives a 2D image corresponding to one text from the text set from the model learning unit 30, and passes the features of the 2D base model that uses the text as input back to the model learning unit 30. The 2D base model is an object detection model that uses text as input, such as Grounding DINO.

[0058] Specifically, for example, the text feature generation unit 50 inputs the text and 2D images received from the model learning unit 30 into a 2D base model trained on a large dataset, and outputs a feature vector in a high-dimensional feature space calculated during processing by this 2D base model, as shown in (20) below.

[0059]

[0060] In the second embodiment, in addition to the learning described in the first embodiment, the model learning unit 30, for example, when text T_j is enabled from the text set, passes this text T_j and the two-dimensional image shown in (21) below, which is based on the two-dimensional image from which the subject of the text has been extracted, to the text feature generation unit 50, and obtains the text features shown in (22) below from the text feature generation unit 50.

[0061]

[0062] Furthermore, the model learning unit 30 passes the viewpoint image shown in (24) and the text T_j, which are rendered from a viewpoint different from the viewpoint of the 2D image I_i to be processed, using so-called different camera parameters corresponding to the 2D image shown in (23) below, to the text feature generation unit 50, and obtains the text features shown in (25) below from this text feature generation unit 50.

[0063]

[0064] The model learning unit 30 calculates the loss related to the text features for different camera parameters, which were obtained as described above, as shown in (26) below.

[0065]

[0066] The loss function can be a function based on the similarity between vectors, such as cosine similarity. The model learning unit 30 adds the loss used to optimize the loss shown in (26) above, and based on this loss, evaluates whether the target defined as valid in the text can be controlled across different viewing angles.

[0067] Figure 8 is a block diagram showing an example of the hardware configuration of a data processing device according to one embodiment of the present invention. In the example shown in Figure 8, the data processing device 100 according to the above embodiment is configured by, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115. The data processing device 100 will be described below, but the same applies to the data processing device 100a shown in Figure 5.

[0068] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0069] The input / output interface 113 is connected to an input device 500 and an output device 600, which are attached to the data processing device 100 and used by users or the like.

[0070] The input / output interface 113 can capture operation data entered by a user or the like through an input device 500 such as a keyboard, touch panel, or touchpad, and can also output output data to an output device 600, including a display device using liquid crystal or organic EL (electroluminescence), for display. The input device 500 and output device 600 may be devices built into the data processing device 100, or they may be input and output devices of other information terminals that can communicate with the data processing device 100 via a network.

[0071] The program memory 111B is a non-temporary tangible storage medium in which a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), is used in combination with another non-volatile memory such as ROM (Read Only Memory), and can store programs necessary for executing various control processes, etc., according to one embodiment.

[0072] The data memory 112 is a tangible storage medium that, for example, uses a combination of the above-mentioned non-volatile memory and volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during the process of various operations.

[0073] A data processing device 100 according to one embodiment of the present invention may be configured as a data processing device having, as a software-based processing function unit, various parts of the data processing device 100, for example, a dataset input unit 10, a guide information creation unit 20, a model learning unit 30, and a model control unit 40 as shown in Figure 1.

[0074] The storage devices used as work memories by each part of the data processing device 100 may be configured using the data memory 112 shown in Figure 8. However, the storage areas configured by these storage devices are not essential to the data processing device 100, and may be, for example, external storage media such as USB (Universal Serial Bus) memory, or areas provided in storage devices such as database servers located in the cloud.

[0075] Each processing function in the data processing device 100 can be implemented by having the hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0076] Furthermore, the methods described in each embodiment can be stored as programs (software means) that can be executed by a computer on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memories, etc.), and can also be transmitted and distributed via communication media. The programs stored on the media also include configuration programs that configure the computer to run software means (including not only the execution program but also tables or data structures). The computer implementing this device reads the program recorded on the recording media and, if necessary, constructs the software means using the configuration program, and executes the above-described processes by controlling the operation of this software means. Note that the recording media referred to in this specification are not limited to those for distribution, but also include storage media such as magnetic disks or semiconductor memories provided inside the computer or in devices connected via a network.

[0077] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0078] 100, 100a...Data processing unit 10...Dataset input unit 20...Guide information creation unit 30...Model learning unit 40...Model control unit 50...Text feature generation unit

Claims

1. A data processing device comprising: a creation unit that creates a set of mask images including a binary mask image corresponding to each of the multiple types of patterns and a binary mask image indicating a region not included in the region indicated by the binary mask image, based on parameters indicating the conditions at which the two-dimensional image to be processed was captured and text information indicating each of the multiple types of patterns of the elements to be reproduced as a three-dimensional space in the two-dimensional image; a learning unit that inputs the two-dimensional image to be processed and parameters indicating the conditions at which the two-dimensional image was captured into a model that takes the two-dimensional image as input and reproduces a three-dimensional space using a Gaussian distribution, obtains a rendering image from the model corresponding to a specified pattern among the multiple types of patterns, obtains a two-dimensional image which is the element product of the two-dimensional image to be processed and the mask image corresponding to the specified pattern from the set of mask images, and learns the model based on the loss between the two-dimensional image and the rendering image; and an image processing unit that inputs a new two-dimensional image to be processed into the learned model to obtain a two-dimensional image corresponding to the specified pattern among the multiple types of patterns.

2. The learning unit inputs a two-dimensional image, which is the element-wise product of a first two-dimensional image to be processed and a mask image corresponding to the specified pattern from the set of mask images, and the text information indicating the specified pattern to a model that takes a two-dimensional image and text information indicating the specified pattern as input and outputs feature quantities of the text information indicating the specified pattern, thereby obtaining first feature quantities of the text information indicating the specified pattern; inputs a second two-dimensional image, which is an image from a viewpoint different from the viewpoint relating to the first two-dimensional image, and parameters indicating the conditions when the second two-dimensional image was taken, to the learned model, thereby obtaining a rendering image corresponding to a specified pattern from among the multiple types of patterns; inputs the rendering image and the text information indicating the specified pattern to a model that outputs feature quantities of the text information, thereby obtaining second feature quantities of the text information indicating the specified pattern; and evaluates a model that reproduces the three-dimensional space based on the loss between the first and second feature quantities, as described in claim 1.

3. A method performed by a data processing device, comprising: a creation unit of the data processing device creating a set of mask images including a binary mask image corresponding to each of the multiple types of patterns and a binary mask image indicating a region not included in the region indicated by the binary mask image, based on parameters indicating the conditions at the time of capturing a two-dimensional image to be processed and text information indicating each of the multiple types of patterns of elements to be reproduced as a three-dimensional space in the two-dimensional image; a learning unit of the data processing device inputting the two-dimensional image to be processed and parameters indicating the conditions at the time of capturing the two-dimensional image to be processed into a model that takes the two-dimensional image as input and reproduces a three-dimensional space using a Gaussian distribution, obtaining a rendering image from the model corresponding to a specified pattern among the multiple types of patterns, obtaining a two-dimensional image which is the element product of the two-dimensional image to be processed and the mask image corresponding to the specified pattern from the set of mask images, and learning the model based on the loss between the two-dimensional image and the rendering image; and an image processing unit of the data processing device inputting a new two-dimensional image to be processed into the learned model to obtain a two-dimensional image corresponding to the specified pattern among the multiple types of patterns.

4. A data processing program that causes a processor to function as a component of the data processing device described in claim 1 or 2.

Images