Information processing method, program, and information processing device

Abstract

This information processing method comprises: acquiring, as first data, a first domain image together with a character string including a special keyword (S11 and S15); calculating a first error between an image of an inference result from an inference model when the character string including the special keyword is input and the first domain image (S19); acquiring, as second data, a second domain image together with a character string corresponding to the second domain image (S21 and S25); calculating a second error between an image of an inference result from the inference model when the character string corresponding to the second domain image is input and the second domain image (S29); and training the inference model by machine learning so as to reduce the sum of errors based on the first error and the second error (S31).

Description

Information Processing Method, Program, and Information Processing Apparatus

[0001] The present invention relates to an information processing method, a program, and an information processing apparatus.

[0002] In various detection systems, image recognition that can automatically perform highly accurate detection from acquired images using AI (Artificial Intelligence) technology has become possible (see, for example, Patent Document 1). When performing image recognition by AI, training data including a large number of image data and labels corresponding to the image data is required.

[0003] Japanese Patent Application Laid-Open No. 2022-128923

[0004] On the other hand, there are cases where appropriate image recognition by AI cannot be performed because a large amount of learning data cannot be obtained. Therefore, the present invention provides an information processing method and the like that can generate training data for appropriately performing image recognition by AI.

[0005] An information processing method according to an aspect of the present invention is an information processing method executed by a computer that trains an inference model that generates an image, the method including: acquiring a first domain image as first data together with a character string including a special keyword; calculating a first error between an image of an inference result of the inference model when the character string including the special keyword is input using the first data and the first domain image; acquiring a second domain image as second data together with a character string corresponding to the second domain image; calculating a second error between an image of an inference result of the inference model when the character string corresponding to the second domain image is input using the second data and the second domain image; and training the inference model by machine learning so as to reduce the sum of the errors based on the first error and the second error.

[0006] A program according to an aspect of the present invention is a program for causing the computer to execute the information processing method described above.

[0007] Furthermore, an information processing device according to one aspect of the present invention is an information processing device for training an inference model that generates images, comprising: a first acquisition unit that acquires a first domain image as first data together with a string containing special keywords; a first calculation unit that uses the first data to calculate a first error between the image of the inference result of the inference model when the string containing the special keywords is input and the first domain image; a second acquisition unit that acquires a second domain image as second data together with a string corresponding to the second domain image; a second calculation unit that uses the second data to calculate a second error between the image of the inference result of the inference model when the string corresponding to the second domain image is input and the second domain image; and a training unit that trains the inference model by machine learning to reduce the sum of the errors based on the first error and the second error.

[0008] These comprehensive or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium.

[0009] The information processing method of the present invention can generate training data for appropriate image recognition by AI.

[0010] Figure 1 is a block diagram showing the configuration of the information processing device in the embodiment. Figure 2 is a flowchart showing the processing of the information processing device in the embodiment. Figure 3 is a flowchart showing the processing of the information processing device in the embodiment. Figure 4 is a flowchart showing the processing of the information processing device in the embodiment. Figure 5 is a conceptual diagram for explaining the processing of the information processing device in the embodiment. Figure 6 is a conceptual diagram for explaining the processing of the information processing device in the embodiment. Figure 7 is a diagram illustrating the data generated by the processing of the information processing device in the embodiment.

[0011] (Knowledge that formed the basis of this invention) The inventors have found that the following problems arise with respect to the AI-based image recognition technology described in the "Background Art" section. Specifically, in AI-based image recognition, when a recognition model trained in an existing environment is deployed to a new environment, the recognition performance may deteriorate due to differences in how images are perceived (acquisition conditions) between environments. To solve this problem, conventionally, new training data for the recognition model is acquired in the new environment, and the recognition model is trained using that data.

[0012] In other words, the effort required to acquire data in a new environment is significant, posing a barrier to deploying AI-based image recognition in that environment.

[0013] The conditions under which images are acquired in an existing environment and a new environment include differences in the configuration of the image acquisition device, such as wavelength and optical element specifications, differences in climate, season, and time of day when the image was acquired, and differences between computer graphics generated by a simulator and real-world images.

[0014] This invention provides an information processing method that enables the training of a recognition model capable of high-precision image recognition even when the acquired data is small (for example, only a few images), in order to reduce the effort required to acquire data in a new environment. This information processing method generates a large amount of training data for the recognition model using only a small amount of data acquired in a new environment, thereby enabling training for appropriate image recognition by AI.

[0015] As a technique for generating large amounts of training data, we utilize image generation AI, which has made remarkable progress in recent years, to generate a large number of images of a novel environment in a simulated manner. However, while the inference models used in image generation AI can generate high-quality images by providing arbitrary text prompts (strings), it is difficult to generate images that simulate a novel environment using the inference models as they are. Therefore, in the information processing method of this invention, the inference model is trained and fine-tuned so that it can generate images that simulate a novel environment simply by inputting text prompts.

[0016] A more detailed overview of the present invention is as follows:

[0017] An information processing method according to a first aspect of the present invention is an information processing method performed by a computer for training an inference model that generates images, wherein a first domain image is acquired as first data together with a string containing special keywords, a first error is calculated between the first domain image and the image obtained from the inference result of the inference model when the string containing the special keywords is input using the first data, a second domain image is acquired as second data together with the string corresponding to the second domain image, a second error is calculated between the second domain image and the image obtained from the inference result of the inference model when the string corresponding to the second domain image is input using the second data, and the inference model is trained by machine learning to reduce the sum of the errors based on the first error and the second error.

[0018] According to the above embodiment, the features of the first domain image can be mapped to a string containing special keywords. Furthermore, the features of the second domain image can be mapped to a string corresponding to the second domain image. Then, the trained inference model can generate an image similar to the second domain image that has the features of the first domain image by using a string that contains special keywords and corresponds to the second domain image. In other words, it becomes possible to generate an image similar to the existing environment (as the second domain) that has the features of the new environment (as the first domain). Thus, it is possible to generate training data for appropriate AI-based image recognition.

[0019] Furthermore, the information processing method according to the second aspect of the present invention is the information processing method described in the first aspect, wherein the inference model is a diffusion model that generates an image from a noise image by a dediffusion process based on an input string, and in the calculation of the first error, an image of the inference result by a dediffusion process based on an input string containing the special keyword is generated from an image obtained by superimposing noise on the first domain image, and in the calculation of the second error, an image of the inference result by a dediffusion process based on an input string corresponding to the second domain image is generated from an image obtained by superimposing noise on the second domain image.

[0020] According to the above embodiment, in an inference model that performs a dediffusion process, it is possible to generate training data necessary for appropriate AI-based image recognition.

[0021] Furthermore, the information processing method according to the third aspect of the present invention is the information processing method described in the first or second aspect, wherein in the calculation of the first error, the first domain image is converted into information in the latent space and used, and in the calculation of the second error, the second domain image is converted into information in the latent space and used, and the first error and the second error are errors in the latent space.

[0022] According to the above embodiment, each image can be treated as information in the latent space, and the inference model can be trained in a way that minimizes errors in the latent space.

[0023] Furthermore, the information processing method according to the fourth aspect of the present invention is the information processing method described in any one of the first to third aspects, wherein in the calculation of the first error, the string containing the special keyword is converted into vector information and used, and in the calculation of the second error, the string corresponding to the second domain image is converted into vector information and used.

[0024] According to the above embodiment, each string can be treated as vector information to train an inference model.

[0025] Furthermore, the information processing method according to the fifth aspect of the present invention is the information processing method described in the fourth aspect, wherein the conversion of a string to vector information is performed by inputting the string into a conversion model, and in the information processing method, the conversion model is further trained by machine learning to reduce the sum of the errors based on the first error and the second error.

[0026] According to the above embodiment, a transformation model can be used when converting a string into vector information. Furthermore, the transformation model can be trained in conjunction with the training of the inference model.

[0027] Furthermore, the information processing method according to the sixth aspect of the present invention is the information processing method described in any one of the first to fifth aspects, wherein the sum of the errors is calculated by multiplying one of the first error and the second error by a coefficient for scaling adjustment with the other, and adding the one multiplied by the coefficient and the other.

[0028] According to the above embodiment, the difference in scale between the first error and the second error can be equalized by multiplying them by a coefficient for scaling, thereby reducing the error and allowing the inference model to be trained.

[0029] Furthermore, the information processing method according to the seventh aspect of the present invention is the information processing method described in any one of the first to sixth aspects, wherein the conditions under which the first domain image and the second domain image are acquired are different.

[0030] According to the above embodiment, it is possible to generate an image similar to the second domain image that has the characteristics of the first domain image, even when the conditions under which the images were acquired are different for the first domain image and the second domain image.

[0031] Furthermore, the program according to the eighth aspect of the present invention is a program that causes a computer to execute the information processing method described in any one of the first to seventh aspects.

[0032] According to the above embodiment, by having a computer execute it, the same effects as the information processing method described above can be achieved.

[0033] Furthermore, an information processing device according to a ninth aspect of the present invention is an information processing device for training an inference model that generates images, comprising: a first acquisition unit that acquires a first domain image as first data together with a string containing special keywords; a first calculation unit that uses the first data to calculate a first error between the image of the inference result of the inference model when the string containing the special keywords is input and the first domain image; a second acquisition unit that acquires a second domain image as second data together with a string corresponding to the second domain image; a second calculation unit that uses the second data to calculate a second error between the image of the inference result of the inference model when the string corresponding to the second domain image is input and the second domain image; and a training unit that trains the inference model by machine learning to reduce the sum of the errors based on the first error and the second error.

[0034] According to the above embodiment, the same effects as the information processing method described above can be achieved.

[0035] The embodiments will be described in detail below with reference to the drawings.

[0036] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0037] (Embodiment) Figure 1 is a block diagram showing the configuration of the information processing device in this embodiment.

[0038] The information processing device 100 is a device that can generate an image of a new environment (hereinafter also referred to as the target domain) from a text prompt by training an inference model using an image and a text prompt.

[0039] As shown in FIG. 1, the information processing apparatus 100 includes functions of an image acquisition unit 210, a resize / crop processing unit 220, an image encoding unit 230, a variational autoencoder NNW (neural network) 240, a noise generation unit 250, a noise superimposition unit 260, a text prompt acquisition unit 310, a text encoding unit 320, a diffusion process processing unit 330, a diffusion model NNW 340, a text encoder NNW 350, a loss calculation unit 360, and an error backpropagation unit 370. Each function of the information processing apparatus 100 is realized by a processor (e.g., a CPU (Central Processing Unit)) executing a predetermined program using a memory. The information processing apparatus 100 is realized, for example, as an information processing server mounted on a server apparatus such as an edge server or a cloud server, receives input of data from an external apparatus or the like network-connected to the server apparatus, and outputs data corresponding to the input data as necessary. The above functions of the information processing apparatus 100 may be provided in one apparatus, or may be distributed and provided in a plurality of apparatuses communicably connected to each other.

[0040] Also, among the above functions, if at least the image acquisition unit 210, the noise generation unit 250, the noise superimposition unit 260, the text prompt acquisition unit 310, the diffusion process processing unit 330, the diffusion model NNW 340, the loss calculation unit 360, and the error backpropagation unit 370 are provided, an image of a new environment can be generated from a text prompt. That is, functions other than these are not essential functions, and the information processing apparatus 100 may be realized without functions other than the essential functions.

[0041] The image acquisition unit 210 is a function of acquiring images used for training an inference model, that is, source domain images of an existing environment and target domain images of a new environment. The image acquisition unit 210 receives, for example, input of images for training an inference model by an administrator of the information processing apparatus 100. Then, the image acquisition unit 210 acquires the received images.

[0042] The resize / crop processing unit 220 has a function of converting the acquired image into a predetermined size. The resize / crop processing unit 220 generates an image of a predetermined size after conversion from the acquired image by performing processes such as changing the aspect ratio of the image and cropping a part of the image.

[0043] The image encoding unit 230 has a function of converting an image in the pixel space into information in the latent space using the variational autoencoder NNW240. The variational autoencoder NNW240 is a neural network model that has been pre-trained and can map the input image to information in the latent space to reduce the amount of information. By converting the image into information in the latent space and using it, the processing amount by the inference model can be reduced.

[0044] The noise generation unit 250 has a function of generating a random noise image having the same size as the above-mentioned predetermined size. The noise image generated by the noise generation unit 250 is superimposed on the image converted by the image encoding unit 230 in the noise superimposition unit 260. That is, since it is necessary to be information that can be superimposed on the information in the latent space, the noise image generated by the noise generation unit 250 is noise information in the latent space.

[0045] The text prompt acquisition unit 310 has a function of acquiring text prompts (character strings) used for training the inference model, that is, text prompts corresponding to source domain images in the existing environment and text prompts corresponding to target domain images in the new environment. The text prompt acquisition unit 310, for example, receives an input of a text prompt for training the inference model by the administrator of the information processing apparatus 100. Then, the text prompt acquisition unit 310 acquires the input text prompt.

[0046] The text prompt corresponding to the source domain image in the existing environment includes, for example, information about the object class for recognition (character information specifying the class) that is labeled on the source domain image in the existing environment. In other words, the object class information included in the text prompt corresponding to the source domain image in the existing environment allows you to specify the object class to recognize when training the recognition model with the simulated image of the new environment that will be ultimately generated.

[0047] Furthermore, the text prompt corresponding to the target domain image of the new environment includes special keywords to simulate such a new environment, for example, when simulating an image of the new environment. It is preferable that the special keywords be strings that do not frequently appear in the prompts of the generating AI. This reduces the possibility that the generating AI might misinterpret the special keywords when they are used to simulate the new environment. As special keywords, it is preferable to use strings that are generally not easily incorporated into prompts in image generation, such as "sks".

[0048] The image acquisition unit 210 and the text prompt acquisition unit 310 have the function of acquiring the target domain image (first domain image) of the new environment and the prompt (string) containing special keywords, and together they can be called the first acquisition unit. Furthermore, the image acquisition unit 210 and the text prompt acquisition unit 310 have the function of acquiring the source domain image (second domain image) of the existing environment and the prompt (string) corresponding to the second domain image, and together they can be called the second acquisition unit.

[0049] The text encoding unit 320 has the function of converting the acquired text prompt into vector information using the text encoder NNW 350. The text encoder NNW 350 is a pre-trained neural network model that can reduce the amount of information by converting the input text prompt into vector information (i.e., an example of a conversion model). However, the text encoder NNW 350 is continuously trained in conjunction with the training of the inference model described later. In other words, the algorithm for converting the text prompt used as input into vector information will also change according to the training of the inference model.

[0050] The diffusion process processing unit 330, together with the diffusion model NNW 340, constitutes an inference model. The diffusion process processing unit 330 generates a denoised inference result image by having the diffusion model NNW 340 execute a dediffusion process based on the input text prompt on the input image with superimposed noise.

[0051] The loss calculation unit 360 has the function (an example of the first calculation unit) of calculating the reconstruction loss as a first error between the original first domain image and the denoised image obtained by a despreading process based on a prompt (string) containing special keywords for the target domain image of the new environment (first domain image). The loss calculation unit 360 also has the function (an example of the second calculation unit) of calculating the scene-specific preservation loss as a second error between the original second domain image and the denoised image obtained by a despreading process based on a prompt (string) corresponding to the second domain image for the source domain image of the existing environment (second domain image). The first and second errors are errors in the latent space calculated between the images in the latent space. The loss calculation unit 360 uses the first and second errors to calculate the sum of the errors. The sum of the errors is calculated by the following formula (1).

[0052]

[0053] In equation (1) above, the first term in square brackets corresponds to the first error, and the second term corresponds to the second error obtained by multiplying it by a coefficient λ for scaling between the first and second errors. However, z in equation (1) above S and z T Each of these is measured by a variational autoencoder x S The latent vector mapped from x, t and t' are the denoising steps, ε θ The noise, τ, is estimated from the diffusion model of parameter θ. θ The first part is the text encoder, ε~N(0,1) and ε′~N(0,1) are the initial noise maps, and p and p S These correspond to the respective errors given by the prompt.

[0054] The error backpropagation unit 370 performs error backpropagation to reduce the sum of errors calculated as described above, thereby training the inference model (in this case, the diffusion model NNW340 and the text encoder NNW350).

[0055] The processing of the information processing system 10 configured as described above will now be explained.

[0056] Figures 2 to 4 are flowcharts showing the processing of the information processing device in this embodiment. Figures 5 and 6 are conceptual diagrams to explain the processing of the information processing device in this embodiment. Figure 7 is a diagram illustrating the data generated by the processing of the information processing device in this embodiment.

[0057] As shown in Figure 2, in step S11, the image acquisition unit 210 acquires a target domain image. For example, as shown in Figure 5, an image of a foggy cityscape, as indicated by the dot hatching in the figure, is acquired as the target domain image.

[0058] In step S12, the resize / crop processing unit 220 converts the acquired image to a predetermined size. As shown in Figure 5, the image is resized and cropped to the same predetermined size as the noise image to be superimposed.

[0059] In step S13, the image encoding unit 230 uses a variational autoencoder (i.e., VAE) NNW240 to convert an image of a predetermined size into information in latent space.

[0060] In step S14, the noise generation unit 250 generates a noise image, and the noise superposition unit 260 superimposes it onto the encoded image. As shown in Figure 5, a random noise image is generated each time and superimposed onto the resized and cropped image. Note that although Figure 5 shows the image in pixel space for illustrative purposes, in reality, the noise superposition unit 260 superimposes the information of the noise image in latent space onto the image that has been converted to latent space information.

[0061] On the other hand, in step S15, the text prompt acquisition unit 310 acquires the text prompt of the target domain. The text prompt of the target domain is a string that includes the special keyword "V*", as shown in Figure 5 as an example, "a photo of a V* urban scene".

[0062] In step S16, the text encoding unit 320 converts the text prompt acquired using the text encoder NNW 350 into vector information. Although Figure 5 illustrates the use of the text prompt as a string for illustrative purposes, in reality, the text encoding unit 320 encodes the text prompt as described above, and the inference model is trained using the encoded text prompt.

[0063] The processes described above from steps S11 to S14 and from steps S15 to S16 may be executed in any order, or they may be executed in parallel.

[0064] In step S17, the image converted into a latent vector is input to the inference model, i.e., the diffusion process processing unit 330 and the diffusion model NNW 340, along with an encoded prompt, and in step S18, the inference result image is generated by removing noise through a dediffusion process.

[0065] As a result, as shown in Figure 5, an image similar to the target domain image converted to a predetermined size before the noise image was superimposed is obtained.

[0066] In step S19, the loss calculation unit 360 calculates the reconstruction loss between the image generated as described above and the target domain image converted to a predetermined size before the noise image was superimposed.

[0067] Furthermore, as shown in Figure 3, in step S21, the image acquisition unit 210 acquires a source domain image. For example, as shown in Figure 6, an image of a cityscape as shown in the figure is acquired as the source domain image.

[0068] In step S22, the resize / crop processing unit 220 converts the acquired image to a predetermined size. As shown in Figure 6, the image is resized and cropped to the same predetermined size as the noise image to be superimposed.

[0069] In step S23, the image encoding unit 230 uses a variational autoencoder (i.e., VAE) NNW240 to convert an image of a predetermined size into information in latent space.

[0070] In step S24, the noise generation unit 250 generates a noise image, and the noise superposition unit 260 superimposes it onto the encoded image. As shown in Figure 6, a random noise image is generated each time and superimposed onto the resized and cropped image. Note that although Figure 6 shows the image in pixel space for illustrative purposes, in reality, the noise superposition unit 260 superimposes the information of the latent space noise image onto the image that has been converted to latent space information.

[0071] On the other hand, in step S25, the text prompt acquisition unit 310 acquires the text prompt of the source domain. The text prompt of the source domain is a string containing "[CLASS]", which is information about the object class to be recognized, as shown in Figure 6 as "a photo of [CLASS] in an urban scene". Here, [CLASS] can be replaced with a word that specifies one or more arbitrary object classes to be recognized in the cityscape, such as persons, cars, bicycles, trucks, buses, trains, etc.

[0072] In step S26, the text encoding unit 320 converts the text prompt acquired using the text encoder NNW 350 into vector information. Although Figure 6 illustrates the use of the text prompt as a string for illustrative purposes, in reality, the text encoding unit 320 encodes the text prompt as described above, and the inference model is trained using the encoded text prompt.

[0073] The processes described above from steps S21 to S24 and from steps S25 to S26 may be executed in any order, or they may be executed in parallel.

[0074] In step S27, the image converted into a latent vector is input to the inference model, i.e., the diffusion process processing unit 330 and the diffusion model NNW 340, along with an encoded prompt, and in step S28, the inference result image is generated by removing noise through a dediffusion process.

[0075] As a result, as shown in Figure 6, an image similar to the source domain image converted to a predetermined size before the noise image was superimposed is obtained.

[0076] In step S29, the loss calculation unit 360 calculates the scene-specific storage loss between the image generated as described above and the source domain image converted to a predetermined size before the noise image was superimposed.

[0077] After steps S19 and S29 are completed, as shown in Figure 4, in step S31, the loss calculation unit 360 multiplies the scene-specific storage loss by a coefficient λ to weight it, and calculates the sum (sum of errors) of the reconstruction loss and the scene-specific storage loss. In this embodiment, as shown in equation (1) above, the reconstruction loss, the scene-specific storage loss and the sum of errors are calculated all at once using a single formula (steps S19, S29 and S31 are performed simultaneously).

[0078] In step S32, the error backpropagation unit 370 trains the inference model (particularly the diffusion model NNW340 and the text encoder NNW350) so that the sum of the calculated errors becomes smaller. The inference model trained in this manner can generate an image that simulates the target domain image by inputting a string that includes special keywords, such as the text prompt "a photo of cars in a V* urban scene," and corresponds to the source domain image, as shown in Figure 7.

[0079] In this way, by using a large number of available source domain images to generate images corresponding to the object class to be recognized, it becomes possible to specify that the features of target domain images (i.e., the differences between the source domain and the target domain), of which only a few images are available, be included in the generated images by including "special keywords (V*)" in the prompt. In other words, even when only a small number of target domain images are available, the inference model can be trained to appropriately generate images that mimic the target domain images. And because a large number of images mimicking the target domain images (training data) can be generated in this way, it becomes possible to easily train the recognition model so that it can be used in new environments.

[0080] (Other Embodiments) Although embodiments have been described above, the present invention is not limited to the embodiments described above.

[0081] For example, in the above embodiment, a process performed by a specific processing unit may be performed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0082] Furthermore, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0083] Furthermore, each component may be implemented by hardware. Each component may also be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or they may be separate circuits. Also, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0084] Furthermore, general or specific embodiments of the present invention may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. Alternatively, they may be implemented as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0085] Furthermore, although the above embodiment described a server device as an example of an information processing device, the information processing device may also be implemented in a way that it is mounted on a general-purpose computer device or the like.

[0086] Furthermore, the present invention may be implemented as an information processing method performed by a computer, such as the information processing device of the above embodiment, or as a program for causing a computer to execute an information processing method. The present invention may also be implemented as a computer-readable, non-temporary recording medium on which such a program is stored.

[0087] Furthermore, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art could conceive, or forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of the present invention.

[0088] This invention can be used to train recognition models for image recognition.

[0089] 100 Information processing unit 210 Image acquisition unit 220 Resize / crop processing unit 230 Image encoding unit 240 Variational autoencoder NNW 250 Noise generation unit 260 Noise superposition unit 310 Text prompt acquisition unit 320 Text encoding unit 330 Diffusion process processing unit 340 Diffusion model NNW 350 Text encoder NNW 360 Loss calculation unit 370 Error backpropagation unit

Claims

1. A computer-based information processing method for training an inference model that generates images, comprising: acquiring a first domain image as first data along with a string containing special keywords; calculating a first error between the image obtained from the inference result of the inference model when the string containing the special keywords is input and the first domain image using the first data; acquiring a second domain image as second data along with a string corresponding to the second domain image; calculating a second error between the image obtained from the inference model when the string corresponding to the second domain image is input and the second domain image using the second data; and training the inference model by machine learning to minimize the sum of the errors based on the first and second errors.

2. The information processing method according to claim 1, wherein the inference model is a diffusion model that generates an image from a noise image by a dediffusion process based on an input string, the first error calculation generates an image of the inference result by a dediffusion process based on an input string containing the special keyword from an image obtained by superimposing noise on the first domain image, and the second error calculation generates an image of the inference result by a dediffusion process based on an input string corresponding to the second domain image from an image obtained by superimposing noise on the second domain image.

3. The information processing method according to claim 1, wherein in the calculation of the first error, the first domain image is converted into information in the latent space and used, and in the calculation of the second error, the second domain image is converted into information in the latent space and used, and the first error and the second error are errors in the latent space.

4. The information processing method according to claim 1, wherein in the calculation of the first error, a string containing the special keyword is converted into vector information and used, and in the calculation of the second error, a string corresponding to the second domain image is converted into vector information and used.

5. The information processing method according to claim 4, wherein the conversion of a string to vector information is performed by inputting the string into a conversion model, and the information processing method further trains the conversion model by machine learning to reduce the sum of the errors based on the first error and the second error.

6. The information processing method according to claim 1, wherein the sum of the errors is calculated by multiplying one of the first error and the second error by a coefficient for scaling adjustment with the other, and adding the one multiplied by the coefficient and the other.

7. The information processing method according to claim 1, wherein the conditions under which the first domain image and the second domain image were acquired are different.

8. A program for causing the computer to execute the information processing method described in any one of claims 1 to 7.

9. An information processing device for training an inference model that generates images, comprising: a first acquisition unit that acquires a first domain image as first data along with a string containing special keywords; a first calculation unit that uses the first data to calculate a first error between the inference result image of the inference model when the string containing the special keywords is input and the first domain image; a second acquisition unit that acquires a second domain image as second data along with a string corresponding to the second domain image; a second calculation unit that uses the second data to calculate a second error between the inference result image of the inference model when the string corresponding to the second domain image is input and the second domain image; and a training unit that trains the inference model by machine learning to reduce the sum of the errors based on the first error and the second error.

Images