Data processing apparatus, data processing method, and data processing program

The data processing device addresses low accuracy in MLLMs by generating training datasets for graph diagrams and captions, enhancing the performance of MLLMs in interpreting and generating captions for graph diagrams.

WO2026239683A1PCT designated stage Publication Date: 2026-11-19HITACHI LTD
0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/001969
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-01-22
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Current MLLMs (Multimodal Large Language Models) have low accuracy in reading graph diagrams due to limitations in image encoder performance, and existing methods require large training datasets that are difficult to obtain, especially for specialized industrial diagrams.

Method used

A data processing device generates training datasets by randomly determining conditions for graph diagrams and captions, using AI to create graph plots and captions, and performs comparative learning on multimodal language models to improve accuracy.

Benefits of technology

Facilitates the generation of training datasets for comparative learning, enhancing the performance of MLLMs in interpreting graph diagrams and improving accuracy in generating captions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026001969_19112026_PF_FP_ABST
    Figure JP2026001969_19112026_PF_FP_ABST
Patent Text Reader

Abstract

This data processing apparatus comprises a processor that executes a program, and a storage device that stores the program. The data processing apparatus stores, in the storage device, drawing input data serving as a generation source of a graph diagram that is composed of nodes and edges. The processor executes a drawing process of drawing the graph diagram when the drawing input data is input, a generation process of generating a caption that is a sentence describing the graph diagram on the basis of the drawing input data, and a process of outputting the graph diagram drawn by the drawing process and the caption generated by the generation process.
Need to check novelty before this filing date? Find Prior Art

Description

Data Processing Device, Data Processing Method, and Data Processing Program Incorporation by Reference

[0001] This application claims the priority of Japanese Application No. 2025-82903 filed on May 16, 2025 (Reiwa 7), the entire content of which is incorporated into this application by reference.

[0002] The present invention relates to a data processing device, a data processing method, and a data processing program for processing data.

[0003] Current MLLMs (Multimodal Large Language Models) have particularly low accuracy in reading "graph diagrams" in which the relationships between symbols and texts are represented by straight lines. The image encoder forms the performance base of MLLMs and RAG (Retrieval Augmented Generation). The image encoder receives an image as input and outputs an image feature vector. The performance of MLLMs and RAG is limited by the performance of the image encoder.

[0004] Non-Patent Document 1 below discloses a technology that uses a deep generative model such as a Diffusion model and an LLM to generate captions and images, and creates a dataset for contrastive learning.

[0005] Hammoud, Hasan Abed Al Kader, et al. "SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?." arXiv preprint arXiv:2402.01832 (2024).

[0006] However, the technology of Non-Patent Document 1 described above requires the preparation of a set of training datasets (diagram images and captions) used for contrastive learning of the image encoder in the deep generative model. Since the performance of the image encoder is limited by the diagram image generation capability of the deep generative model, a large amount of training datasets is still required for its improvement. In addition, since specialized diagrams used in industry are rarely disclosed on the Internet, it is difficult to prepare a sufficient amount of training datasets required for contrastive learning of the image encoder.

[0007] The present invention aims to facilitate the generation of training datasets used for controlled learning.

[0008] A data processing device that represents one aspect of the invention disclosed in this application is a data processing device having a processor that executes a program and a storage device that stores the program, wherein the storage device stores drawing input data which is the source for generating a graph diagram composed of nodes and edges, and the processor is characterized in that it performs a drawing process that draws the graph diagram when the drawing input data is input, a generation process that generates a caption which is a sentence describing the graph diagram based on the drawing input data, and a process that outputs the graph diagram drawn by the drawing process and the caption generated by the generation process.

[0009] According to a typical embodiment of the present invention, it is possible to facilitate the generation of training datasets used for comparative learning. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

[0010] Figure 1 is a block diagram showing an example of the hardware configuration of a computer. Figure 2 is a block diagram showing an example of the functional configuration of a data processing device. Figure 3 is an explanatory diagram showing example of a control learning dataset 1. Figure 4 is an explanatory diagram showing example of a control learning dataset 2. Figure 5 is an explanatory diagram showing an example of a display screen. Figure 6 is a flowchart showing a detailed example of the processing procedure for generating a control learning dataset by the data processing device. Figure 7 is an explanatory diagram showing an example of control learning by the control learning unit. Figure 8 is an explanatory diagram showing an example of weight setting by the control learning unit. Figure 9 is a flowchart showing a detailed example of the processing procedure for control learning by the control learning unit. Figure 10 is a block diagram showing an example of the functional configuration of a learning device. Figure 11 is an explanatory diagram showing example of a control learning dataset 3. Figure 12 is a block diagram showing an example of the functional configuration of an evaluation device.

[0011] <Figure 1: Example of Hardware Configuration of Data Processing Device> Figure 1 is a block diagram showing an example of the hardware configuration of a data processing device. The data processing device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, storage device 102, input device 103, output device 104, and communication IF 105 are connected by a bus 106. The processor 101 controls the data processing device 100. The storage device 102 serves as the work area for the processor 101. The storage device 102 is also a non-temporary or temporary recording medium that stores various programs and data. Examples of storage devices 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Input device 103 receives data. Examples of input devices 103 include a keyboard, mouse, touch panel, numeric keypad, scanner, microphone, and sensor. Output device 104 outputs data. Examples of output devices 104 include a display, printer, and speaker. Communication IF 105 connects to the network and sends and receives data.

[0012] <Figure 2: Example of Functional Configuration of Data Processing Device 100> Figure 2 is a block diagram showing an example of the functional configuration of the data processing device 100. The data processing device 100 includes a drawing input data generation unit 201, a drawing unit 202, a caption generation unit 203, an output unit 204, and a comparison learning unit 205. Specifically, the drawing input data generation unit 201, the drawing unit 202, the caption generation unit 203, the output unit 204, and the comparison learning unit 205 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 shown in Figure 1.

[0013] Furthermore, the data processing device 100 includes a drawing input DB 210, a graph diagram DB 220, and a caption DB 230. Specifically, the drawing input DB 210, the graph diagram DB 220, and the caption DB 230 are implemented, for example, by the storage device 102 shown in Figure 1.

[0014] [Drawing Input Data Generation Unit 201] The drawing input data generation unit 201 generates drawing input data. Drawing input data is input data that is input to the drawing unit 202 to draw a graph diagram. A graph diagram is a diagrammatic image that shows a directed graph such as a circuit diagram, flowchart, sequence diagram, state transition diagram, or class diagram.

[0015] When generating a circuit diagram as a graph using the drawing unit 202, the drawing input data generation unit 201 uses a script to randomly determine conditions such as three resistors, two power supplies, and no closed loops. Based on the determination results, the drawing input data generation unit 201 uses a script to generate a circuit diagram file (for example, an .asc file) as drawing input data to be input into electronic circuit simulation software such as LTspice.

[0016] Furthermore, the drawing input data generation unit 201 may output a prompt to the generation AI (Artificial Intelligence) requesting the generation of a circuit diagram file according to the above-described conditions, and as a result, obtain the circuit diagram file from the generation AI. The generation AI may be implemented within the data processing device 100, or it may be implemented in another data processing device that is communicatively connected to the data processing device 100.

[0017] When the drawing unit 202 is to generate a graph other than a circuit diagram, the drawing input data generation unit 201 randomly determines the graph generation conditions using a script, and based on the determination result, generates a graph code to be input to graph generation software such as Mermaid as drawing input data.

[0018] A graph diagram is a diagram composed of nodes and edges, such as directed graphs like flowcharts, sequence diagrams, state transition diagrams, and class diagrams. Circuit diagrams are also a type of graph diagram. The conditions for generating a graph diagram, such as the number of nodes, the direction of the edges, and the shape of the nodes, are determined randomly by a script.

[0019] The drawing input data generation unit 201 assigns pre-prepared labels to the nodes. A label is a string that indicates the name of the node or the processing content that the node will perform. For example, labels such as "START" and "Check Temperature" are set for the nodes.

[0020] The drawing input data generation unit 201 sets directed edges between nodes with a predetermined probability. The drawing input data generation unit 201 also randomly sets the color scheme of the nodes and edges.

[0021] The drawing input data generation unit 201 generates Mermaid codes indicating the labeled nodes and directed edges obtained in this manner as drawing input data.

[0022] Alternatively, the drawing input data generation unit 201 may output the following prompt to the generating AI, and as a result, obtain a Mermaid code from the generating AI.

[0023] "Create a CI / CD pipeline flowchart using Mermaid. Requirements: * Graph TD * 8-12 nodes * After Build, branch into Test_A and Test_B in parallel, and merge into Deploy after both are completed * Includes a conditional branch to Rollback (Yes / No) * Code blocks should contain only Mermaid code."

[0024] Alternatively, the drawing input data generation unit 201 may randomly select one of several pre-prepared Mermaid code samples, output a prompt to the generation AI as shown below, and obtain the Mermaid code from the generation AI as a result.

[0025] "Randomly change the order of processes or add steps in the following Mermaid code to generate a new Mermaid code. However, the output should only include a description of the flowchart. Mermaid code: (Sample of randomly selected Mermaid code)."

[0026] The drawing input data generated by the drawing input data generation unit 201 is stored in the drawing input DB 210. The drawing input DB 210 may also contain manually collected circuit diagram files and Mermaid codes. In the drawing input DB 210, each piece of drawing input data is assigned a data ID that uniquely identifies the drawing input data.

[0027] [Drawing Unit 202] The drawing unit 202 implements graph plotting software and draws a graph plot when drawing input data is input to the graph plotting software. The graph plotting software is software that draws a graph plot according to the drawing input data when drawing input data is input, such as electronic circuit simulation software like LTspice or graph plot generation software like Mermaid. Specifically, for example, when a circuit diagram file is input as drawing input data, the drawing unit 202 draws the circuit diagram as a graph plot, and when Mermaid code is input as drawing input data, it draws the graph plot as a graph plot. The graph plot is, for example, an SVG file or a PNG file.

[0028] The graphs drawn by the drawing unit 202 are stored in the graph diagram DB 220. In the graph diagram DB 220, each graph is assigned a graph diagram ID that uniquely identifies the graph, and the data ID of the drawing input data from which it was generated is associated with it.

[0029] [Caption Generation Unit 203] The caption generation unit 203 generates captions based on the drawing input data. Captions are text that describe the information of the figure in the graph. If the drawing input data is a circuit diagram file, the caption generation unit 203 reads the bill of materials and wiring from the circuit diagram file using a pre-prepared parser and converts them into captions written in a circuit description language such as SPICE netlist, captions written in a graph description language such as Mermaid format or DOT format, or captions written in natural language such as "V1 is in series with R1".

[0030] Furthermore, if the drawing input data is a graph code such as Mermaid code, the caption generation unit 203 will use the graph code itself as the caption. Alternatively, the caption generation unit 203 may, for example, output a prompt to the generation AI instructing it to convert the graph code into natural language, and obtain natural language such as "In this flowchart..." from the generation AI and use it as the caption.

[0031] The captions generated by the caption generation unit 203 are stored in the caption database 230. In the caption database 230, each caption is assigned a unique caption ID, and the data ID of the drawing input data from which it was generated is associated with it.

[0032] [Output Unit 204] The output unit 204 outputs graphs stored in the graph DB 220 and captions stored in the caption DB 230. Specifically, for example, the output unit 204 outputs a graph and a caption having a data ID associated with that graph. The output unit 204 may display the graph and caption on a display, which is an example of an output device 104, print the graph and caption to a printer, which is an example of an output device 104, or transmit the graph and caption to another data processing device via the communication IF 205. The other data processing device displays the received graph and caption.

[0033] [Comparative Learning Unit 205] The comparative learning unit 205 has a multimodal language model and performs comparative learning on the multimodal language model using a comparative learning dataset which is a combination of graphs and captions having the same data ID.

[0034] A multimodal language model is a language model that interprets and translates modalities such as images, audio, and video, in addition to text. Specifically, for example, a multimodal language model can generate a descriptive text for an image when it is input. Details of contrastive learning will be discussed later.

[0035] <Figure 3 Example of Controlled Learning Dataset 1> Figure 3 is an explanatory diagram showing Example of Controlled Learning Dataset 1. Figure 3 shows a controlled learning dataset 300 when the drawing input data is a circuit diagram file. The controlled learning dataset 300 has a graph 301 showing a circuit diagram and a caption 302 explaining the circuit diagram shown in the graph 301.

[0036] <Figure 4 Example of a comparative learning dataset 2> Figure 4 is an explanatory diagram showing an example of a comparative learning dataset 2. Figure 4 shows a comparative learning dataset 400 when the drawing input data is a graph diagram code. The comparative learning dataset 400 has a graph diagram 401 and a caption 402 that explains the graph diagram 401.

[0037] <Figure 5 Example of Display Screen> Figure 5 is an explanatory diagram showing an example of a display screen. The display screen 500 is shown on a display, which is an example of an output device 104.

[0038] The display screen 500 includes a setting area 501, a graph display area 502, a caption display area 503, a save button 504, and an erase button 505.

[0039] The setting area 501 is an area that accepts input for setting conditions through user operation. The setting area 501 includes a first checkbox 511, a second checkbox 512, a condition input field 513, and an execution button 514.

[0040] The first checkbox 511 is a user interface that accepts a specification for automatic generation of a reference learning dataset.

[0041] The second checkbox 512 is a user interface that accepts a specification of conditions for drawing input data.

[0042] The data processing apparatus 100 accepts selection of only one of the first checkbox 511 and the second checkbox 512.

[0043] The condition input field 513 is a user interface that accepts input of conditions for drawing input data (for example, in the case of a circuit diagram, the number of resistors, the number of power supplies, the presence or absence of a closed loop, etc.). The conditions input into the condition input field 513 are valid when a check mark is input into the second checkbox 512.

[0044] The execution button 514 is a user interface that accepts a start of execution of automatic generation of the reference learning dataset when pressed. When the execution button 514 is pressed in a state where a check mark is input into the first checkbox 511, the data processing apparatus 100 generates drawing input data by the drawing input data generation unit 201, generates a graph diagram by the drawing unit 202 based on the generated drawing input data, and generates a caption by the caption generation unit 203 based on the generated drawing input data.

[0045] When the execution button 514 is pressed in a state where a check mark is input into the second checkbox 512, the data processing apparatus 100 generates drawing input data under the conditions input into the condition input field 513 by the drawing input data generation unit 201, generates a graph diagram by the drawing unit 202 based on the generated drawing input data, and generates a caption by the caption generation unit 203 based on the generated drawing input data.

[0046] The graph diagram display area 502 is an area for displaying a graph diagram. Specifically, for example, when the execution button 514 is pressed, the graph diagram generated by the drawing unit 202 is displayed. In FIG. 5, a graph diagram 301 is displayed as an example.

[0047] The caption display area 503 is an area for displaying captions. Specifically, for example, when an execution button 514 is pressed, the caption generated by the caption generation unit 203 is displayed. In FIG. 5, a caption 302 is displayed as an example.

[0048] A save button 504 is a user interface for saving the generated contrast learning dataset to the storage device 102 when pressed. In FIG. 5, when the save button 504 is pressed, the contrast learning dataset 300 is saved to the storage device 102. Specifically, for example, the data processing apparatus 100 may newly register the contrast learning dataset 300 in a database not shown. Further, the data processing apparatus 100 may set a flag indicating that the graph 301 is selected as the contrast learning dataset 300 in the graph DB 220, and set a flag indicating that the caption 302 is selected as the contrast learning dataset 300 in the caption DB 230.

[0049] A delete button 505 is a user interface for excluding the generated contrast learning dataset from saving targets when pressed. In FIG. 5, when the delete button 505 is pressed, the contrast learning dataset 300 is excluded from the saving targets in the storage device 102, and is deleted from the graph display area 502 and the caption display area 503.

[0050] Specifically, for example, the data processing apparatus 100 may delete the graph 301 that is the contrast learning dataset 300 from the graph DB 220, and delete the caption 302 that is the contrast learning dataset 300 from the caption DB 230. Further, the data processing apparatus 100 may not set the flag indicating that the graph 301 is selected as the contrast learning dataset 300 in the graph DB 220, and may not set the flag indicating that the caption 302 is selected as the contrast learning dataset 300 in the caption DB 230.

[0051] <Figure 6: Controlled Learning Dataset Generation Process> Figure 6 is a flowchart showing a detailed example of the processing procedure for generating a controlled learning dataset by the data processing device 100.

[0052] (Step S601) The data processing device 100 determines whether the control learning dataset generation process has been started. Specifically, for example, the data processing device 100 determines whether the execution button 514 has been pressed. The data processing device 100 waits until the control learning dataset generation process has been started (Step S601: No). If it has been started (Step S601: Yes), the process proceeds to Step S602.

[0053] (Step S602) The data processing device 100 generates drawing input data using the drawing input data generation unit 201.

[0054] (Step S603) The data processing device 100, using the drawing unit 202, performs a drawing process to draw a graph based on the drawing input data.

[0055] (Step S604) The data processing device 100, using the caption generation unit 203, performs a caption generation process to generate a caption based on the drawing input data.

[0056] (Step S605) The data processing device 100 performs an association process to associate the graph diagram generated in step S603 with the caption generated in step S604. Specifically, for example, the data processing device 100 stores the graph diagram in the graph diagram DB 220 along with the data ID of the drawing input data from which it was generated, and stores the caption in the caption DB 230 along with the data ID of the drawing input data from which it was generated.

[0057] (Step S606) The data processing device 100 performs output processing to output the graph generated in step S603 and the caption generated in step S604. For example, as shown in Figure 5, the data processing device 100 displays the graph 301 in the graph display area 502 and displays the caption 302 in the caption display area 503.

[0058] <Figure 7: Example of comparative learning by the comparative learning unit 205> Figure 7 is an explanatory diagram showing an example of comparative learning by the comparative learning unit 205. The comparative learning unit 205 includes an image encoding unit 701, a text encoding unit 702, a similarity calculation unit 703, and a learning processing unit 704. The image encoding unit 701 and the text encoding unit 702 constitute a multimodal language model.

[0059] The image encoding unit 701 reads each of the graphs 710-1 to 710-n in the graph group 710 and generates image feature vectors I1 to In that represent the features of graphs 710-1 to 710-n. The image encoding unit 701 is composed of, for example, a neural network. Hereinafter, n is an integer of 2 or greater, and i in graph 710-i and image feature vector Ii is an integer satisfying 1 ≤ i ≤ n.

[0060] The text encoding unit 702 reads each of the captions 720-1 to 720-n from the caption group 720 and generates text feature vectors T1 to Tn that represent the characteristics of the captions 720-1 to 720-n. The text encoding unit 702 is composed of, for example, a neural network. In the caption 720-j and the text feature vector Tj, j is an integer satisfying 1 ≤ j ≤ n.

[0061] The combination of graph 710-i and caption 720-i constitutes one set of comparative learning datasets. Furthermore, the image encoding unit 701 may have a dimensionality reduction function to match the dimensionality of the image feature vectors I1 to In with the dimensionality of the text feature vectors T1 to Tn.

[0062] The similarity calculation unit 703 calculates the similarity IiTj (for example, cosine similarity) between the image feature vector Ii and the text feature vector Tj (where j is an integer satisfying 1 ≤ j ≤ n). The similarity may also be the reciprocal of the Euclidean distance between the image feature vector Ii and the text feature vector Tj. In this case, a numerical operation (for example, adding a constant to the denominator) is performed so that the denominator of the reciprocal does not become zero. In any case, the calculation method does not matter as long as the similarity between the image feature vector Ii and the text feature vector Tj is calculated. The range of the similarity is normalized to be between 0.0 and 1.0.

[0063] The similarity calculation unit 703 generates a similarity matrix 730 by calculating the similarity IiTj by brute-force matching the image feature vector Ii and the text feature vector Tj.

[0064] The learning processing unit 704 performs comparative learning on the image encoding unit 701 and the text encoding unit 702. Specifically, for example, the learning processing unit 704 learns that the values ​​of the diagonal components I1T1, ..., IiTj, ..., InTn of the similarity matrix 730 are higher than the values ​​of the off-diagonal components.

[0065] In other words, the learning processing unit 704 repeatedly performs loss calculation, gradient calculation, error backpropagation, and parameter updates for the image encoding unit 701 and the text encoding unit 702. After a predetermined number of epochs, or when the loss falls below a threshold, the learning processing unit 704 terminates the process.

[0066] <Figure 8: Example of weight setting by the comparative learning unit 205> Figure 8 is an explanatory diagram showing an example of weight setting by the comparative learning unit 205. In Figure 8, for the sake of simplicity, the similarity matrix 730 is described as a 3x3 matrix.

[0067] In addition to comparative learning processes (calculation of loss, calculation of gradient, backpropagation, and parameter update), the learning processing unit 704 calculates the similarity between graphs. Specifically, for example, the learning processing unit 704 calculates the similarity between graphs. The similarity between graphs is called graph similarity. For example, the Jaccard coefficient between edge sets is used for graph similarity, but it may be calculated by other methods.

[0068] Specifically, for example, the learning processing unit 704 calculates graph similarity for all graphs 710-1 to 710-3. The graph similarity between identical graphs 710-1 and 710-1, between graphs 710-2 and 710-2, and between graphs 710-3 and 710-3 will be the maximum value of "1".

[0069] In the case of graphs 710-1 to 710-3 shown in Figure 8, the graph similarity between identical graphs 710-1 and 710-1 is weighted w11, the graph similarity between identical graphs 710-2 and 710-2 is weighted w22, and the graph similarity between identical graphs 710-3 and 710-3 is weighted w33.

[0070] Furthermore, the graph similarity between different graphs 710-1 and 710-2 is weighted w12 and w21, the graph similarity between different graphs 710-1 and 710-3 is weighted w13 and w31, and the graph similarity between different graphs 710-2 and 710-3 is weighted w23 and w32.

[0071] The learning processing unit 704 sets the weight wij to the similarity IiTj and updates the similarity IiTj to similarity wij・IiTj. In the example in Figure 8, for the diagonal components, similarity I1T1 is updated to similarity w11・I1T1, similarity I2T2 is updated to similarity w22・I2T2, and similarity I3T3 is updated to similarity w33・I3T3.

[0072] Furthermore, for off-diagonal components, the similarity I1T2 is updated to similarity w12・I1T2, similarity I1T3 is updated to similarity w13・I1T3, similarity I2T1 is updated to similarity w21・I2T1, similarity I2T3 is updated to similarity w23・I2T3, similarity I3T1 is updated to similarity w31・I3T1, and similarity I3T2 is updated to similarity w32・I3T2.

[0073] In this way, by weighting the similarity IiTj of the similarity matrix 730 with the graph similarity wij, the similarity of the structures between graphs is taken into account in the similarity IiTj, thereby improving the accuracy of comparative learning.

[0074] <Figure 9 Comparative Learning Process> Figure 9 is a flowchart showing a detailed example of the comparative learning process performed by the comparative learning unit 205.

[0075] (Step S901) The control learning unit 205 acquires a group of control learning datasets. Specifically, for example, the control learning unit 205 reads graphs and captions having the same data ID from the graph DB 220 and caption DB 230, respectively, as a set of control learning datasets. The control learning unit 205 repeatedly performs this reading process for each data ID to acquire a group of control learning datasets, which are multiple sets of control learning datasets.

[0076] (Step S902) The comparison learning unit 205 performs image encoding processing using the image encoding unit 701. Specifically, for example, the comparison learning unit 205 converts each of the graphs 710-1 to 710-n in the graph group 710 of the comparison learning dataset into image feature vectors I1 to In.

[0077] (Step S903) The comparative learning unit 205 performs text encoding processing using the text encoding unit 702. Specifically, for example, the comparative learning unit 205 converts each of the captions 720-1 to 720-n in the caption group 720 in the comparative learning dataset into text feature vectors T1 to Tn.

[0078] (Step S904) The comparison learning unit 205 performs a similarity calculation process using the similarity calculation unit 703. Specifically, for example, the comparison learning unit 205 calculates the similarity IiTj by brute-force matching the image feature vectors I1 to In and the text feature vectors T1 to Tn, and generates a similarity matrix 730.

[0079] (Step S905) The comparison learning unit 205, using the learning processing unit 704, calculates the graph similarity between the graphs, sets the weight wij for each element of the similarity matrix 730, and updates the similarity matrix 730.

[0080] (Step S906) The comparative learning unit 205 performs comparative learning using the similarity matrix 730 updated in step S905 by the learning processing unit 704. Specifically, for example, the learning processing unit 704 uses the updated similarity matrix 730 to calculate the loss L1(τ) related to the direction from the graph to the caption and the loss L2(τ) related to the direction from the caption to the graph, and calculates the average of both losses L1 and L2 (hereinafter, loss L(τ)). τ is a temperature parameter.

[0081] Prior to backpropagation, the learning processing unit 704 calculates various gradients. Specifically, for example, the learning processing unit 704 calculates the derivative of the loss L, ∂L / ∂(sij), for each element of the updated similarity matrix 730. sij is the similarity obtained by normalizing the elements wij・IiTj of the updated similarity matrix 730 by the temperature parameter τ (see equation (1) below).

[0082] sij=wij・IiTj / τ...(1)

[0083] By partially differentiating sij with respect to Ii, we obtain ∂sij / ∂Ii, and by partially differentiating sij with respect to Tj, we obtain ∂sij / ∂Tj. Due to the chain rule between the derivative of loss L ∂L / ∂(sij) and ∂sij / ∂Ii, we obtain the gradient ∂L / ∂Ii, and due to the chain rule between the derivative of loss L ∂L / ∂(sij) and ∂sij / ∂Tj, we obtain the gradient ∂L / ∂Tj. Furthermore, by partially differentiating loss L with respect to the temperature parameter τ, we obtain the gradient ∂L / ∂τ.

[0084] The learning processing unit 704 updates the parameters in the image encoding unit 701 by backpropagating errors using the gradient ∂L / ∂Ii, updates the parameters in the text encoding unit 702 by backpropagating errors using the gradient ∂L / ∂Tj, and updates the temperature parameter τ by backpropagating errors using the gradient ∂L / ∂τ.

[0085] The learning processing unit 704 terminates step S906 when it has updated the parameters a predetermined number of times or when the loss L falls below a predetermined value. In this way, the image encoding unit 701 and the text encoding unit 702 are trained.

[0086] In the example above, the similarity IiTj of the similarity matrix 730 was weighted by the graph similarity wij, but to simplify the process, it is also possible to set it so that the graph similarity wij is not weighted.

[0087] Thus, according to Example 1, a control learning dataset can be easily generated.

[0088] Example 2 describes an example of training a multimodal language model that generates captions from graphs using a control training dataset. In Example 2, the differences from Example 1 will be explained, and the common parts with Example 1 will be omitted from the explanation.

[0089] <Figure 10 Example of Functional Configuration of Data Processing Device> Figure 10 is a block diagram showing an example of the functional configuration of a data processing device. The data processing device 1000 includes an image encoding unit 701, a text encoding unit 702, a text decoding unit 1001, and a learning unit 1002. The image encoding unit 701, the text encoding unit 702, and the text decoding unit 1001 constitute a multimodal language model. Specifically, the text decoding unit 1001 and the learning unit 1002 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 shown in Figure 1.

[0090] In Figure 10, the text decoding unit 1001 generates output text 1012 based on the input text 1011 and graph 710-i, using the control learning dataset 1070-ij (graph 710-i, caption 720-j). The input text 1011 is a prompt requesting the generation of a caption describing graph 710-i. The learning unit 1002 determines how well the output text 1012 predicts caption 820-j and trains the text decoding unit 1001 accordingly.

[0091] The text decoding unit 1001 constitutes part of the multimodal language model. Specifically, for example, the text decoding unit 1001 takes the text feature vector Tk1 of the input text 1011 and the image feature vector Ii of graph figure 710-i as input, and generates a caption describing graph figure 710-i as output text 1012. The output text 1012 is converted into a text feature vector Tk2 by the text encoding unit 702.

[0092] The learning unit 1002 takes the text feature vector Tj of the caption 720-i and the text feature vector Tk2 of the output text 1012 as input and learns the text decoding unit 1001. Specifically, for example, the learning unit 1002 repeats the following operations a predetermined number of times or until the loss is less than or equal to a predetermined value: loss calculation between the caption 720-i and the output text 1012, gradient calculation, and parameter updates within the text decoding unit 1001 by backpropagation using the gradient.

[0093] This improves the prediction accuracy of the output text 1012 obtained from graph 710-i, and makes it possible to predict the caption 720-j from graph 710-i.

[0094] Example 3 shows an example of caption generation. Example 3 describes an example of generating a caption from graph 401 shown in Figure 4. Note that in Example 3, the differences from Examples 1 and 2 will be explained, and the common parts with Examples 1 and 2 will not be explained.

[0095] <Figure 11 Example of Controlled Learning Dataset 3> Figure 11 is an explanatory diagram showing Example of Controlled Learning Dataset 3. The caption 402 explaining the graph 401 shown in Figure 4 describes the definitions of nodes and edges in the order of node "A", edge "->", and node "B", as in "A->B". In Figure 11, in order to facilitate the learning of the image encoding unit 701, the text encoding unit 702, and the text decoding unit 1001, the caption generation unit 203 converts caption 720-j from caption 402 to caption 1102 prior to inputting to the text encoding unit 702, so that it is node ⇒ node ⇒ ... ⇒ edge, that is, the definition of the node is described first, followed by the definition of the edge, and no definition of the node is described after that.

[0096] First, the caption generation unit 203 extracts nodes A, B, and C and edges (A, B) and (A, C) from the caption 402. Even if the content of caption 720-j is natural language, the caption generation unit 203 extracts the names that become nodes and the connecting phrases that become edges according to the named entity rules.

[0097] Next, the caption generation unit 203 arranges the extracted nodes A, B, and C in unique order, and if the content of caption 720-j is in Mermaid format as caption 402, it writes "Graph TD; A B C", and if it is in natural language format, it writes "Node: A, B, C".

[0098] Next, the caption generation unit 203 appends the original connection information as is. Specifically, for example, if the content of caption 720-j is in Mermaid format like caption 402, it appends the original connection information "A->B A->C", and if it is in natural language format, it appends the original connection information "Connection: A→B, A→C". In this way, caption 720-j is converted from caption 402 to caption 1102.

[0099] Caption 1102 is written in the order of node ⇒ edge, and after the edge is written, the node is not written again.

[0100] The data processing device 100 performs comparative learning using graph 401 and caption 1102 as the comparative learning dataset 400, which makes learning easier compared to using caption 402. In other words, predicting only existing nodes is easier than predicting edges, thus improving the accuracy of inference and making learning easier.

[0101] Example 4 describes an example of evaluating the recognition capabilities of a multimodal language model. In Example 4, the differences from Examples 1 to 3 will be explained, and the common parts with Examples 1 to 3 will be omitted from the explanation.

[0102] <Figure 12 Example of Functional Configuration of Data Processing Device> Figure 12 is a block diagram showing an example of the functional configuration of a data processing device. The data processing device 1200 includes an output unit 204, an image encoding unit 701, a text encoding unit 702, a text decoding unit 1001, an analysis unit 1201, and a similarity calculation unit 1202. Specifically, the analysis unit 1201 and the similarity calculation unit 1202 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 shown in Figure 1.

[0103] The image encoding unit 701 encodes graph figure 710-i to generate an image feature vector Ii. The text encoding unit 702 encodes the input text 1011 to generate a text feature vector Tk3. The input text 1011 is a prompt requesting the generation of a caption describing graph figure 710-i.

[0104] The text decoding unit 1001 receives the image feature vector Ii from the image encoding unit 701 and generates a caption explaining graph figure 710-i as output text 1012.

[0105] The analysis unit 1201 is an existing parser that converts text into graph information. Graph information is information that defines the components of a graph diagram, such as nodes in the graph diagram and edges connecting nodes. The analysis unit 1201 analyzes the output text 1012 from the text decoding unit 1001 and converts the output text 1012 into graph information 1220.

[0106] The similarity calculation unit 1202 calculates the graph similarity between the ground truth graph information 1210-i and the graph information 1220. The ground truth graph information 1210-i is graph information prepared in advance for graph figure 710-i. The graph similarity may be calculated using, for example, the Jaccard coefficient between edge sets, but it may also be calculated by other methods.

[0107] The output unit 204 outputs the graph similarity calculated by the similarity calculation unit 1202. The output unit 204 may also output the graph figure 710-i, the correct graph information 1210-i, the output text 1012, and the graph information 1220 along with the graph similarity. The output unit 204 may display this data on a display, which is an example of an output device 104, print it to a printer, which is another example of an output device 104, or transmit it to another computer via the communication IF 205.

[0108] Thus, according to Example 4, the user can evaluate the recognition capability of the multimodal language model by referring to graph similarity.

[0109] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all the configurations described above. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with other configurations.

[0110] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0111] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0112] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A data processing device comprising a processor that executes a program and a storage device that stores the program, wherein the storage device stores drawing input data which is the source for generating a graph diagram composed of nodes and edges, and the processor executes: a drawing process which draws the graph diagram when the drawing input data is input; a generation process which generates a first caption which is a sentence describing the graph diagram based on the drawing input data; and a process which outputs the graph diagram drawn by the drawing process and the first caption generated by the generation process.

2. A data processing device according to claim 1, wherein the processor performs an association process that associates a graph drawn by the drawing process with a first caption generated by the generation process and stores it in the storage device.

3. A data processing device according to claim 1, wherein the graph is a circuit diagram.

4. A data processing device according to claim 1, wherein the graph is a directed graph.

5. A data processing device according to claim 4, wherein the directed graph is a flowchart.

6. A data processing device according to claim 1, wherein the first caption contains the definition of the edge following the definition of the node, and does not contain the definition of the node following the definition of the edge.

7. A data processing device according to claim 1, wherein in the generation process, the processor generates the first caption such that it describes the definition of the node, describes the definition of the edge following the definition of the node, and does not describe the definition of the node following the definition of the edge.

8. A data processing device according to claim 2, wherein the processor performs: an image encoding process in which, for each combination of the graph and the first caption associated by the association process, the image encoding unit encodes the graph constituting the combination to generate an image feature vector; a text encoding process in which, for each combination, the text encoding unit encodes the first caption constituting the combination to generate a text feature vector; and a learning process in which the image encoding unit and the text encoding unit perform comparative learning based on the similarity between the image feature vector generated for each combination by the image encoding process and the text feature vector generated for each combination by the text encoding process.

9. A data processing device according to claim 8, wherein in the learning process, the processor calculates the graph similarity between two graphs in the set of combinations for each of the two graphs, and performs comparative learning of the image encoding unit and the text encoding unit based on the similarity between the image feature vector and the text feature vector and the graph similarity.

10. A data processing device according to claim 1, wherein the processor performs: an image encoding process in which an image encoding unit encodes the graph and generates an image feature vector; a text decoding process in which a text decoding unit decodes the image feature vector and generates a second caption; and a learning process in which the text decoding unit learns based on the loss between the first caption and the second caption generated by the text decoding process.

11. A data processing device according to claim 1, wherein the storage device stores first graph information defining the components of the graph diagram, and the processor performs: an image encoding process by an image encoding unit to encode the graph diagram and generate an image feature vector; a text decoding process by a text decoding unit to decode the image feature vector and generate a second caption; an analysis process to analyze the second caption generated by the text decoding process and generate second graph information defining the components of the graph diagram; and a calculation process to calculate the graph similarity between the first graph information and the second graph information generated by the analysis process.

12. A data processing method to be executed by a data processing device having a processor that executes a program and a storage device that stores the program, wherein the storage device stores drawing input data which is the source for generating a graph diagram composed of nodes and edges, and the processor executes: a drawing process which draws the graph diagram when the drawing input data is input; a generation process which generates a caption which is a sentence describing the graph diagram based on the drawing input data; and a process which outputs the graph diagram drawn by the drawing process and the caption generated by the generation process.

13. A data processing program characterized by causing a processor to execute: a drawing process that draws a graph when drawing input data which is the source of a graph composed of nodes and edges is input; a generation process that generates a caption which is a sentence describing the graph based on the drawing input data; and a process that outputs the graph drawn by the drawing process and the caption generated by the generation process.