Data generation system

Abstract

A data generation system constructed by a training system, said data generation system: comprising a decoder (24) and a latent space (23) for intermediate data z of the training system; being configured such that, when first data sampled from intermediate data in the latent space is input to the decoder, the decoder generates and outputs output data similar to input data during training; and further comprising a data feature extraction unit (22, 32) that can extract sampling points of intermediate data that characterize the distribution of output data x' generated by the decoder.

Description

Data Generation System Cross-Reference to Related Applications

[0001] This international application claims the benefit of Japanese Patent Application No. 2025-068884 filed with the Japan Patent Office on April 18, 2025, the entire disclosure of which is incorporated into this international application by reference.

[0002] The present disclosure relates to a data generation system.

[0003] As a machine learning model, a variational autoencoder (hereinafter referred to as VAE) including an encoder and a decoder is known. In the VAE, the encoder and the decoder are trained such that the output from the decoder matches or is similar to the input to the encoder. Further, after training, if first data of intermediate data in the latent space between the encoder and the decoder is input to the decoder, the decoder can generate output data similar to the input data used during training. Therefore, a data generation system can be constructed by using the latent space and the decoder of the trained VAE.

[0004] In a data generation system using a VAE, as described in, for example, Patent Document 1, it has been proposed to allow a user to grasp the meaning of the latent space by adding the meaning of each dimension in the latent space using a disentanglement representation by β-VAE. Note that VAE is an abbreviation for Variational Auto Encoder, and the disentanglement representation refers to a representation with disentangled features.

[0005] Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2024-500730

[0006] The VAE regards the data distribution of intermediate data as a standard normal distribution, and generates data to be input to the decoder from the standard normal distribution. According to the proposed apparatus described above, by strongly approximating the intermediate data to the normal distribution, the independence of each coordinate axis (that is, each dimension) of the data distribution is increased, and the meaning of each coordinate axis can be observed.

[0007] However, the proposed device had a problem in that the constraint of making the data distribution closer to a standard normal distribution was too strong, resulting in a deterioration of the approximation accuracy between the data distribution and the standard normal distribution, as well as the accuracy of data reconstruction caused by making the data distribution closer to a normal distribution. For this reason, the proposed device had a problem in that it was difficult to generate output data that corresponded to the data structure of the input data during training.

[0008] One aspect of this disclosure is to provide a data generation system that enables the observation of data structure and the generation of data based on that observation.

[0009] One aspect of the data generation system of this disclosure includes a latent space of intermediate data and a decoder, which are constructed by a learning system that includes an encoder that converts input data into intermediate data and a decoder that generates output data from the intermediate data, and the output data is learned to match the input data.

[0010] Furthermore, the data generation system disclosed herein is configured such that when first data sampled from intermediate data in the latent space is input to a decoder, the decoder generates and outputs output data similar to the input data used during training.

[0011] Furthermore, the data generation system disclosed herein includes a data feature extraction unit configured to extract sampling points of intermediate data that characterize the generation distribution of output data by the decoder.

[0012] Therefore, according to the data generation system disclosed herein, not only is it possible to generate data, but it is also possible to analyze the data distribution of the output data generated by the decoder using the sampling points of the intermediate data extracted by the data feature extraction unit. Thus, the user can generate desired output data based on the analysis results.

[0013] This is a block diagram showing the overall configuration of the data generation system of the embodiment. This is an explanatory diagram showing the data generation model of the embodiment. This is an explanatory diagram showing the data generation model using VAE. This is an explanatory diagram showing the data generation model using Vamp Prior VAE. This is an explanatory diagram showing the evaluation results of the data generation models of the embodiment, VAE, and Vamp Prior VAE. This is an explanatory diagram explaining the method for analyzing the data distribution in the data generation model of the embodiment. This is an explanatory diagram showing a modified example of the data generation distribution learning unit of the embodiment.

[0014] Embodiments of this disclosure will be described below with reference to the drawings.

[0015] [Configuration of the Data Generation System] As shown in Figure 1, the data generation system 2 of this embodiment includes an arithmetic processing unit 10, which is a computer including a CPU, ROM, RAM, etc. Peripheral devices such as an input unit 12, an output unit 14, a communication control unit 16, and a storage unit 18 are connected to the arithmetic processing unit 10.

[0016] Here, the input unit 12 is implemented using an input device such as a keyboard or mouse, and in response to input operations by the operator, it inputs various instruction information, such as the start of processing, to the arithmetic processing unit 10. The output unit 14 is implemented using a display device such as a liquid crystal display, a printing device such as a printer, etc. The communication control unit 16 is implemented using a NIC (i.e., a Network Interface Card), etc., and controls communication between the arithmetic processing unit 10 and external devices such as a server via a network.

[0017] Next, the storage unit 18 is implemented using a storage device such as a non-volatile semiconductor memory, hard disk, or optical disc. The storage unit 18 stores a processing program for operating the arithmetic processing unit 10 as a data generation system, as well as the data generation model 20 shown in Figure 2.

[0018] Furthermore, the memory unit 18 also stores pseudo-input data u generated during the construction of the data generation model 20 and data generation commands t corresponding to the pseudo-input data u, as information that characterizes the generation distribution of output data x' by the data generation model 20. This will be explained later.

[0019] The arithmetic processing unit 10 generates predetermined data, such as image data, using the data generation model 20 stored in the storage unit 18, in accordance with input commands received from the input unit 12. The generated data is then output to the output unit 14 or stored in the storage unit 18 as output data. The arithmetic processing unit 10 can also transmit the output data generated by the data generation model 20 to an external device via the communication control unit 16.

[0020] Furthermore, the arithmetic processing unit 10 also functions as a learning unit 30 for constructing the data generation model 20. The generation of output data using the data generation model 20 is performed after the learning operation by the learning unit 30 is completed and the data generation model 20 has been constructed. In addition, the data generation model 20 constructed by the learning operation by the learning unit 30 can be used by other data generation systems by transmitting it to an external device via the communication control unit 16 from the arithmetic processing unit 10.

[0021] [Data Generation Model 20] The data generation model 20 of this embodiment is an extension of the Vamp Prior VAE data generation model, which is based on the VAE data generation model, and, like each of these data generation models, has an encoder 22 and a decoder 24.

[0022] Therefore, first, the data generation model 20A using VAE and the data generation model 20B using Vamp Prior VAE, which are the basic configurations of the data generation model 20 of this embodiment, will be explained using Figures 3 and 4.

[0023] The encoder 22 and decoder 24 are each composed of neural networks. The encoder 22 converts input data x into intermediate data z, and the decoder 24 generates output data x' from the intermediate data z.

[0024] In the VAE data generation model 20A shown in Figure 3, the encoder 22 and decoder 24 are trained so that the output data x' matches the input data x. Therefore, the VAE data generation model 20A shown in Figure 3 functions as the learning system of this disclosure.

[0025] In other words, in the VAE data generation model 20A, the encoder 22 receives, for example, tens of thousands of input data points x as training data for the data generation model 20A. The training of the data generation model 20A is carried out so that the data distribution of the intermediate data z in the latent space 23 between the encoder 22 and the decoder 24 follows a standard normal distribution.

[0026] Furthermore, during data generation after this learning process, the data distribution of the intermediate data z in the latent space 23 is considered to be a standard normal distribution, and random numbers are generated according to an external data generation command to sample one of the intermediate data z in the latent space 23.

[0027] The sampled intermediate data is then input to the decoder 24 as the first data for data generation, causing the decoder 24 to generate (i.e., reconstruct) the data, and the generated data is output from the decoder 24 as output data x'.

[0028] Furthermore, as a data generation model utilizing VAE, as described in Patent Document 1 above, it has also been proposed that the meaning of each dimension in the latent space 23 is added using a disentanglement representation by β-VAE, so that the user can grasp the meaning of the latent space.

[0029] However, in this proposal, as well as in the data generation model 20A using VAE, the constraint of making the data distribution closer to a standard normal distribution is too strong, which leads to a deterioration in the approximation accuracy between the data distribution and the standard normal distribution, as well as the reconstruction accuracy caused by making the data distribution closer to a normal distribution.

[0030] On the other hand, a Vamp Prior VAE is known as a VAE that solves this problem, extending the data distribution of the latent space 23 as a mixture distribution (for example, a mixture of Gaussian distributions) that includes components provided by a variational posterior distribution conditioned on a learnable pseudo-input.

[0031] As shown in Figure 4, in the Vamp Prior VAE data generation model 20B, k pseudo-input data u (i.e., u1 to uk) are created in the same input space as the input data x of the VAE data generation model 20A shown in Figure 3. Typically, this number k is set to be smaller than the number of input data x.

[0032] Then, by inputting each pseudo-input data u into the encoder 22, the k normal distributions obtained, along with the normal distribution of the input data x, are used to approximate the data distribution of the intermediate data z in the latent space 23 with a two-layer mixed normal distribution.

[0033] This data generation model 20B uses a mixed normal distribution with higher severity than the VAE-based data generation model 20A shown in Figure 3, thereby improving both the approximation accuracy of the data distribution of intermediate data z with respect to input data x and the reconstruction accuracy by the decoder 24.

[0034] Note that VAEs are publicly known technologies, as described in works such as Diederik P. Kingma and Max Welling's "Auto-Encoding Variational Bayes" (https: / / arxiv.org / pdf / 1312.6114). Similarly, Vamp Prior VAEs are also publicly known technologies, as described in works such as Jakub M. Tomczak and Max Welling's "VAE with a VampPrior" (https: / / arxiv.org / pdf / 1705.07120). For this reason, a detailed explanation of the data generation models 20A and 20B shown in Figures 3 and 4 will be omitted.

[0035] Incidentally, the Vamp Prior VAE data generation model 20B shown in Figure 4 can achieve high reconstruction accuracy by increasing the expressive power of the data distribution in the latent space 23 through the use of a mixture of normal distributions, but the data distribution of the intermediate data z does not take the form of a normal distribution.

[0036] Therefore, unlike the VAE-based data generation model 20A shown in Figure 3, this data generation model 20B cannot obtain a disentanglement representation in which each dimension in the latent space 23 is separated according to the factors and properties in the input data x, making it difficult to analyze the data structure.

[0037] Therefore, in the data generation model 20B, the user cannot use the results of the data structure analysis to grasp the sampling points of the intermediate data z that characterize the generation distribution of the output data x' by the decoder 24, and thus cannot generate the desired data.

[0038] Therefore, the data generation model 20 of this embodiment shown in Figure 2 is an extension of the data generation model 20B using Vamp Prior VAE shown in Figure 4, and is configured to generate pseudo-input data u using an embedding map 26.

[0039] In other words, in the data generation model 20 of this embodiment, instead of directly providing pseudo-input data u to the data space of the input data x to the encoder 22, as in Vamp Prior VAE, the pseudo-input data u is provided using the embedding map 26.

[0040] Here, the embedding map 26 is a map of embedded representations in which data such as images is converted into numerical vectors that are easy for computers to handle, and semantically similar concepts are placed in close proximity.

[0041] As is well known, this embedding map 26 can represent data as a sequence of numbers, for example, "0.2, 0.5, ...", and by inputting such a sequence of numbers as a data generation command, it can output data corresponding to the generation command.

[0042] The pseudo-input data u generated in the embedding map 26 is then embedded in the data space of the encoder 22 and input to the encoder 22.

[0043] That is, the data generation model 20 of the present embodiment includes a data generation distribution learning unit 32 that is provided with an embedding map 26 and a data generation command input unit 28 that inputs a data generation command to the embedding map 26.

[0044] Then, in the data generation distribution learning unit 32, the data generation command input unit 28 inputs k pieces of binary data to the embedding map 26 as data generation commands t, so as to generate k pieces of pseudo input data u. Note that the k data generation commands t are stored in the storage unit 18.

[0045] As described above, this pseudo input data u is embedded in the data space on the input side to the encoder 22 (that is, the input space), input to the encoder 22, and converted into intermediate data. Then, the data generation distribution learning unit 32 learns the embedding map 26 such that the distribution of the converted intermediate data z becomes a normal distribution, and the pseudo input data u matches the output data x' from the decoder 24.

[0046] That is, the embedding map 26 is configured by a neural network different from the encoder 22 and the decoder 24, and in the data generation distribution learning unit 32, learning is performed so that the data distribution of the intermediate data z converted by the encoder 22 becomes a plurality of normal distributions.

[0047] Here, for the learning of the embedding map 26, amortized inference which is also used in VAE is employed, and the output u corresponding to the input t to the embedding map 26 is not learned independently, but is learned in consideration of all the inputs t. Further, during the learning of the embedding map 26, the pseudo input data u obtained by the embedding map 26 is input to the encoder 22 together with the input data x, and converted into the corresponding normal distribution by amortized inference.

[0048] Therefore, even if the input data x to the encoder 22 and the output data x' from the decoder 24 are different, when they are converted to the same intermediate data z by the encoder 22, the result will be the same data. Consequently, the space in which the input t is transformed by the embedding map 26 is not large, and is large enough to represent the transformed space by the encoder 22. This reduces the difficulty of the search, in other words, it leads to improved accuracy and reduced computation time.

[0049] Through the above learning process, similar to Vamp Prior VAE, the data distribution of the intermediate data z in the latent space 23 can be approximated by a two-layer mixed normal distribution, thereby improving the approximation accuracy and reconstruction accuracy of the data distribution of the intermediate data z.

[0050] Furthermore, the data generation distribution learning unit 32 adjusts the entire pseudo-input data u using the embedding map 26, rather than individually adjusting the pseudo-input data u as in Vamp Prior VAE.

[0051] Therefore, the data generation model 20 of this embodiment can improve the approximation accuracy of the data distribution of intermediate data z and the reconstruction accuracy of output data x' compared to the data generation model 20B which uses Vamp Prior VAE.

[0052] In this embodiment, the embedding map 26 functions as a pseudo-input data generation unit of the present disclosure. The encoder 22 and data generation distribution learning unit 32 constituting the data generation model 20 function as a data feature extraction unit of the present disclosure. The sampling points of the intermediate data that characterize the generation distribution of the output data extracted by the data feature extraction unit of the present disclosure are identified by the data generation command t and pseudo-input data u stored in the storage unit 18.

[0053] [Approximate Accuracy and Reconstruction Accuracy] To confirm this effect, the reconstruction error and KL information were evaluated in the data generation model 20 of this embodiment, the VAE-based data generation model 20A, and the Vamp Prior VAE-based data generation model 20B.

[0054] The reconstruction error was evaluated using the squared error between the input data x to the encoder 22 and the output data x' from the decoder 24. The KL divergence is the Kullback-Leibler divergence, used in machine learning to compare two probability distributions. Furthermore, to compare the evaluation of each data generation model 20, 20A, and 20B, encoders 22 and decoders 24 with a common structure were used.

[0055] Figure 5 shows the evaluation results, with the horizontal axis representing KL divergence and the vertical axis representing reconstruction error. Two datasets, "mnist" and "fmnist," were used for this evaluation.

[0056] In Figure 5, the performance improves as you move diagonally downwards to the left, indicating that the data generation model 20 of this embodiment has superior performance, i.e., approximation accuracy and reconstruction accuracy, compared to the other data generation models 20A and 20B.

[0057] Furthermore, in this embodiment, by making the number of pseudo-input data u k less than the number of input data x N (i.e., k ≪ N), the computational cost during training can be reduced, and overfitting of the intermediate data z to the input data x and pseudo-input data u can be suppressed.

[0058] [Data Distribution Analysis] In Figure 6, Figure 6A shows a method for analyzing the data distribution when the dimension L of the data input space to the encoder 22 is set to "4" and the data generation command t is set to 4-bit binary data, thereby generating pseudo-input data u in the embedding map 26.

[0059] Furthermore, Figure 6B illustrates a method for analyzing the data distribution when the dimension L of the data input space to the encoder 22 is set to "9" and the data generation command t is set to 9-bit binary data, thereby generating pseudo-input data u in the embedding map 26.

[0060] As shown in the upper part of Figure 6A, when the data generation command t is 4-bit binary data, 16 different data generation commands t, ranging from "0000" to "1111", are input to the embedding map 26.

[0061] Therefore, for example, if the pseudo-input data u generated by the embedding map 26 is a handwritten digit "2", the embedding map 26 will output 16 types of character data corresponding to the data generation command t as pseudo-input data u. This pseudo-input data u is then stored in the storage unit 18 by the data generation distribution learning unit 32 in accordance with the data generation command t.

[0062] Therefore, the user can obtain a data generation command t for the data generation model 20 to generate output data x' corresponding to the stored pseudo-input data u by outputting this stored pseudo-input data u to a display or other device that constitutes the output unit 14.

[0063] In other words, the user can understand what kind of pseudo-input data u will be generated by setting the values ​​of the four nodes of the data generation command t, which corresponds to each bit of the binary data, to "1" or "0".

[0064] Furthermore, since the data generation model 20 is trained so that the output data x' matches the pseudo-input data u, it is also possible to understand what shape the output data x' will take from the values ​​of each node in the data generation command t.

[0065] For example, you can view the image of pseudo-input data u1 when all of the four nodes in the 2x2 grid of the data generation command t shown in Figure 6A are set to "0", and the image of pseudo-input data u2 when only the top-left node is set to "0" and the remaining nodes are set to "1". Similarly, you can also view the image of pseudo-input data u3 when the two nodes in the bottom row are set to "0" and the two nodes in the top row are set to "1".

[0066] Furthermore, the embedding map 26 can change the generated pseudo-input data u into an image similar to each numerical value by changing the numerical values ​​of the four nodes that constitute the data generation command t between 1 and 0.

[0067] Therefore, as illustrated in the lower part of Figure 6A, the user can input data generation commands t corresponding to the pseudo-input data u2 and u3 into the embedding map 26, and by gradually changing the value of the "1" node in the data generation command t to "0", they can confirm the change in the output data x'.

[0068] On the other hand, Figure 6B shows a data generation command t when the dimension L of the data input space to the encoder 22 is "9", represented by a 3x3 node arrangement. In Figure 6B, one of the nine nodes is selected as the node of interest and gradually changes from "1" to "0", showing the image change of the output data x'.

[0069] The upper part of Figure 6B shows the image change of the output data x' when the lower left node is selected as the node of interest for the data generation command t, and the value of this node of interest is gradually changed from "1" to "0", while the values ​​of the other nodes are fixed at "0.5".

[0070] Furthermore, the lower part of Figure 6B shows the image change of the output data x' when the rightmost node in the second row is selected as the node of interest for the data generation command t, the value of this node of interest is gradually changed from "1" to "0", and the values ​​of the other nodes are fixed at "0.5".

[0071] Thus, with the data generation system 2 of this embodiment, the user can understand the meaning of each node in the data generation command t by changing the data generation command t and observing the changes in the output data x'. For example, in Figure 6B, it can be seen that setting the lower left node to 1 generates a simple, horizontally wide "2" image, while setting the middle right node to 1 generates a vertically elongated "2" image. Furthermore, setting both of these nodes to 1 creates a simple, vertically elongated "2" image as expected.

[0072] Therefore, according to the data generation system 2 of this embodiment, the data distribution of the output data x' can be analyzed by using the data generation command t during learning of the embedding map 26 stored in the memory unit 18 and the pseudo-input data u corresponding to this data generation command t.

[0073] Furthermore, the user can generate desired output data x' by creating a data generation command t based on the analysis results and inputting it into the embedding map 26 of the data generation model 20 via the input unit 12.

[0074] Furthermore, in this case, the data generation command t can adjust the numerical values ​​of each node in the data generation command t from the binary data used during the training of the embedding map 26 to any value between 0 and 1 or greater than 1.

[0075] In other words, the user can generate arbitrary values, which are new generation commands not included in the binary data (which are generation commands used during training), and input them into the embedding map 26. Therefore, the user can fine-tune the output data x' generated by the data generation system 2.

[0076] [Other Embodiments] Although embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above and can be implemented in various modified forms.

[0077] For example, in the above embodiment, the data generation command input unit 28 was described as inputting k pre-set binary data as data generation commands t to the embedding map 26. However, the data generation command input unit 28 may be configured to generate k random numbers and input them to the embedding map 26, for example, as shown in Figure 7.

[0078] Even in this manner, the embedding map 26 can be trained in the same way as in the above embodiment. After training, the k random numbers input to the embedding map 26 by the data generation command input unit 28, along with the pseudo-input data u corresponding to those random numbers, can be stored in the storage unit 18. In other words, in this way, the user can analyze the data distribution of the output data x' using the random numbers and pseudo-input data u stored in the storage unit 18, and obtain the same effects as in the above embodiment.

[0079] Next, in the above embodiment, the data generation system 2 is configured using one computer that constitutes the arithmetic processing unit 10. However, the multiple functions implemented by the computer may be implemented by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, they may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured with one or more hardware logic circuits. Furthermore, the computer program that implements each of the above functions may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. Moreover, the method for implementing each of the above functions does not necessarily have to include software, and all of the functions may be implemented using one or more hardware components.

[0080] Furthermore, multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, some parts of the configuration of the above embodiment may be omitted. Furthermore, at least some parts of the configuration of the above embodiment may be added to or replaced with the configuration of other above embodiments.

[0081] Furthermore, the technology disclosed herein can be implemented in various forms, including not only a data generation system, but also a system that uses a data generation system as a component, a program for making a computer function as a data generation system, a non-transitional physical recording medium such as semiconductor memory that records this program, and a data generation method.

Claims

1. A data generation system comprising a learning system constructed by a learning system which includes an encoder (22) that converts input data (x) into intermediate data (z) and a decoder (24) that generates output data (x') from the intermediate data, and the system which is trained so that the output data matches the input data, and further comprising a data feature extraction unit (22, 32) configured to extract sampling points of the intermediate data that characterize the distribution of output data generated by the decoder.

2. A data generation system according to claim 1, wherein the data feature extraction unit comprises the encoder of the learning system and a data generation distribution learning unit (32) which learns to bring the data distribution of the pseudo-input data closer to the data distribution of the output data by inputting pseudo-input data of a predetermined data generation distribution to the encoder.

3. A data generation system according to claim 2, wherein the data generation distribution learning unit comprises: a pseudo-input data generation unit (26) configured to generate the pseudo-input data as data input to a data input space for constituting the data generation distribution; and a data generation command input unit (28) configured to input a generation command to the pseudo-input data generation unit to cause it to generate the pseudo-input data of the data generation distribution, and is configured to learn the pseudo-input data generation unit so that the data distribution of the pseudo-input data approaches the data distribution of the output data.

4. A data generation system according to claim 3, wherein the generation command that the data generation command input unit inputs to the pseudo-input data generation unit is binary data.

5. A data generation system according to claim 3 or claim 4, wherein the data feature extraction unit is configured to output, when the data generation distribution learning unit completes the learning operation of the pseudo-input data generation unit, the generation command at the time of the learning operation and the pseudo-input data generated by the pseudo-input data generation unit by the generation command as the extraction results of sampling points of the intermediate data that characterize the generation distribution of the output data.

6. A data generation system according to claim 5, wherein the data generation command input unit has an output data generation function that, after the completion of the learning operation by the data generation distribution learning unit, inputs one generation command to the pseudo-input data generation unit in accordance with an external input command, causing the decoder to output the output data corresponding to the generation command.

7. A data generation system according to claim 6, wherein the data generation command input unit is configured to generate new generation commands that are not included in the data from the generation commands input to the pseudo-input data generation unit during the learning operation by the data generation distribution learning unit, in accordance with the input commands, and to input them to the pseudo-input data generation unit.

Images