Information processing program, information processing method, and information processing device

Abstract

An information processing device (100) acquires at least one first label (111) obtained by analyzing image information (110). The information processing device (100) acquires at least one second label (121) and at least one first edge (122), which are obtained by analyzing document information (120). On the basis of the result of associating any of the first labels (111) with any of the second labels (121), the information processing device (100) identifies at least one label (131) for discerning nodes forming a diagram (101). On the basis of the at least one label (131), the information processing device (100) identifies at least one edge (132) forming the diagram (101) from among the acquired at least one first edge (122).

Description

Information processing program, information processing method, and information processing apparatus

[0001] The present invention relates to an information processing program, an information processing method, and an information processing apparatus.

[0002] Conventionally, there are cases where it is desired for a computer to identify the structure of a diagram including a plurality of nodes and a plurality of edges each connecting the nodes. The diagram is, for example, a hierarchy diagram, a flowchart, a sequence diagram, or the like. For example, a method of identifying the structure of a diagram by analyzing image information of the diagram with a computer is conceivable.

[0003] As a prior art, for example, there is a technique that identifies the row and column to which each object belongs, based on whether each of a plurality of pairs of objects constitutes a component specified by a column and a row of a table, shares the same row, and shares the same column.

[0004] International Publication No. 2023 / 188362

[0005] However, with the conventional technology, it is difficult to accurately identify the structure of a diagram. For example, regarding image information, there is a problem that when the start point or end point of an edge is unclear, or edges overlap each other, it is difficult to accurately identify the edge. Further, for example, regarding image information, when the resolution is relatively low, there is a problem that it is difficult to accurately identify the label of a node.

[0006] In one aspect, an object of the present invention is to improve the accuracy of identifying the structure of a diagram.

[0007] According to one embodiment, an information processing program, information processing method, and information processing device are proposed that obtain one or more first labels that identify nodes included in a diagram by analyzing image information of a diagram, obtain one or more second labels that identify nodes included in a diagram and one or more first edges included in a diagram by analyzing document information of a diagram, identify one or more labels that identify nodes forming the diagram based on the result of associating any of the obtained first labels with any of the obtained second labels, and identify one or more edges that form the diagram from among the obtained first edges based on the identified one or more labels.

[0008] According to one embodiment, it becomes possible to improve the accuracy of identifying the structure of a diagram.

[0009] Figure 1 is an explanatory diagram showing one embodiment of the information processing method according to the embodiment. Figure 2 is an explanatory diagram showing an example of the information processing system 200. Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. Figure 4 is a block diagram showing an example of the functional configuration of the information processing device 100. Figure 5 is a block diagram showing a specific example of the functional configuration of the information processing device 100. Figure 6 is an explanatory diagram (part 1) showing an example of a diagram. Figure 7 is an explanatory diagram (part 2) showing an example of a diagram. Figure 8 is an explanatory diagram (part 1) showing an example of raw data and document information. Figure 9 is an explanatory diagram (part 2) showing an example of raw data and document information. Figure 10 is an explanatory diagram (part 3) showing an example of raw data and document information. Figure 11 is an explanatory diagram (part 4) showing an example of raw data and document information. Figure 12 is an explanatory diagram showing the operation flow of the information processing device 100. Figure 13 is an explanatory diagram showing an example of acquiring image information and document information. Figure 14 is an explanatory diagram (part 1) showing an example of the operation of the information processing device 100. Figure 15 is an explanatory diagram (part 2) showing an example of the operation of the information processing device 100. Figure 16 is an explanatory diagram showing an example of associating semantic information with an edge. Figure 17 is a flowchart showing an example of the overall processing procedure.

[0010] Embodiments of the information processing program, information processing method, and information processing apparatus according to the present invention will be described in detail below with reference to the drawings.

[0011] (An Embodiment of the Information Processing Method According to the Embodiment) Figure 1 is an explanatory diagram showing an embodiment of the information processing method according to the embodiment. The information processing device 100 is a computer for identifying the structure of a diagram. The information processing device 100 is, for example, a server or a PC (Personal Computer).

[0012] Diagrams include, for example, system diagrams, flowcharts, or sequence diagrams. Diagrams include, for example, multiple nodes and multiple edges. Nodes may be labeled. Labels are, for example, information indicating the attributes of the node. Edges connect nodes. Edges may be connected to the same node at both ends.

[0013] Traditionally, it has been desirable to use a computer to identify the structure of a diagram. Specifically, the structure describes what nodes and edges the diagram contains, and which nodes each edge connects to. In particular, one might consider using a computer to evaluate a design document based on the results of identifying the structure of the diagram described in the document.

[0014] In contrast, a first method can be considered in which the structure of a diagram is identified by analyzing the image information of the diagram using a computer. Specifically, the image information refers to the image in which the diagram is drawn. Specifically, it is possible to identify the structure of the diagram by analyzing the image information using a large-scale language model. Specifically, it is possible to identify the structure of the diagram by analyzing the image information using a dedicated detector for analyzing image information. For details on the first method, please refer to Reference 1 below.

[0015] Reference 1: Fang, Jiaqi, Zhen Feng, and Bo Cai. “DrawnNet: offline hand-drawn diagram recognition based on keypoint prediction of aggregating geometry ” Entropy 24.3 (2022): 425.

[0016] The first method has the problem of difficulty in accurately identifying the structure of the diagram. For example, with respect to image information, if the start or end points of edges are unclear, edges overlap, the tips of edges are not arrowhead-shaped, or edges are dotted lines, it may be difficult to accurately identify the edges that form the diagram. Also, for example, the larger the image size, the more difficult it becomes to analyze the entire image together, making it difficult to accurately identify the edges that form the diagram. Furthermore, for example, with respect to image information, if the resolution is relatively low, it may be difficult to accurately identify the labels of the nodes that form the diagram. Also, for example, it may be difficult to accurately identify the labels of the nodes that form the diagram depending on the input limits of the large-scale language model. Input limits are, for example, the upper limit of the amount of data in the image information. Input limits may also be, for example, the upper limit of the overall size, height, or width of the image indicated by the image information.

[0017] On the other hand, a second method can be considered, for example, to identify the structure of a diagram by analyzing the document information of the diagram using a computer. Specifically, the document information refers to the document in which the diagram is described. The document information may be in XML (Extensible Markup Language) format, draw.io format, or astah format, for example. Specifically, it is possible to analyze the document information using a rule-based approach. For more details on this second method, please refer to Reference 2 below.

[0018] Reference 2: Shiinoki, Shue, et al. “Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions.” arXiv preprint arXiv:2502.04389 (2025).

[0019] The second method also has the problem of difficulty in accurately identifying the structure of the diagram. For example, the document information may contain information other than the structure of the diagram. Specifically, the document information may contain information that indicates a legend for the diagram. Specifically, the document information may contain information attached to the diagram, such as the user who created or updated the diagram or the date and time. In this case, it is difficult to accurately identify the nodes that make up the diagram.

[0020] Thus, traditionally, it has been difficult to accurately identify the structure of a diagram. For example, it is difficult to comprehensively and accurately identify what nodes and edges a diagram contains, and which nodes each edge connects to.

[0021] Therefore, this embodiment describes an information processing method that can improve the accuracy of recognizing the structure of a diagram. According to this information processing method, at least the labels that identify the nodes forming the diagram and the edges that form the diagram can be identified with high accuracy.

[0022] In Figure 1, the information processing device 100 identifies the structure of the diagram 101. The diagram 101 includes one or more nodes and one or more edges. Nodes may, for example, be labeled. Edges may, for example, connect nodes. Edges may, for example, have both ends connected to the same node.

[0023] (1-1) The information processing device 100 obtains one or more first labels 111 obtained by analyzing the image information 110 of the diagram 101. The image information 110 shows an image in which the diagram 101 is drawn. The one or more first labels 111 identify, for example, different nodes included in the diagram 101. The information processing device 100 obtains one or more first labels 111 by, for example, analyzing the image information 110. As a result, the information processing device 100 can identify nodes that have a relatively high probability of being components forming the diagram 101.

[0024] (1-2) The information processing device 100 obtains one or more second labels 121 and one or more first edges 122 obtained by analyzing the document information 120 of the diagram 101. The document information 120 indicates the document in which the diagram 101 is described. One or more second labels 121 identify, for example, different nodes included in the diagram 101. One or more first edges 122 are included in the diagram 101. As a result, the information processing device 100 can identify first edges 122 that have a relatively high likelihood as components forming the diagram 101. The information processing device 100 can identify second labels 121 that have a relatively high likelihood as strings.

[0025] (1-3) The information processing device 100 associates one or more acquired first labels 111 with one or more acquired second labels 121. For example, the information processing device 100 associates one or more acquired first labels 111 with one or more acquired second labels 121 whose similarity is above a threshold. As a result, the information processing device 100 can obtain a guideline for identifying one or more labels 131 that identify different nodes forming the diagram 101.

[0026] The information processing device 100 identifies one or more labels 131 that identify different nodes forming the diagram 101, based on the matching results. Here, it is assumed that the nodes identified by the first label 111, which are recognized as part of the diagram 101 when the image information 110 is analyzed and the first label 111 is obtained, are likely to be components of the diagram 101. On the other hand, it is assumed that one or more second labels 121 obtained by analyzing the document information 120 are likely to be strings.

[0027] The information processing device 100 identifies one or more labels 131 that identify different nodes forming the diagram 101 by performing the process (A) or (B) below, in accordance with these assumptions.

[0028] (A) The information processing device 100 corrects one or more first labels 111 that identify different nodes that are likely to be components of the diagram 101 based on one or more second labels 121 that are likely to be strings. The information processing device 100 identifies the one or more corrected first labels 111 as one or more labels 131 that identify different nodes that form the diagram 101.

[0029] The information processing device 100 corrects any of the first labels 111 based on any of the second labels 121 that are associated with the first label 111, among one or more second labels 121.

[0030] Specifically, the information processing device 100 determines whether any of the first labels 111 match any of the second labels 121 associated with that first label 111. If the information processing device 100 determines that they do not match, it corrects the first label 111 to match the most likely second label 121, and identifies the corrected first label 111 as label 131. If the information processing device 100 determines that they do match, it identifies the first label 111 as label 131 as is.

[0031] (B) The information processing device 100 removes one or more second labels 121 that are likely to be strings and that do not correspond to any of the first labels 111 that identify nodes likely to be components of the diagram 101. The information processing device 100 identifies the one or more second labels 121 that remain after removal, each corresponding to a different node likely to be a component of the diagram 101, as one or more labels 131.

[0032] As a result, the information processing device 100 can appropriately combine the results of analyzing the image information 110 and the results of analyzing the document information 120 to accurately identify one or more labels 131 that identify different nodes forming the diagram 101. The information processing device 100 can obtain a guideline for identifying one or more edges 132 that form the diagram 101.

[0033] (1-4) Based on the identified label 131, the information processing device 100 identifies one or more edges 132 that form the diagram 101 from among the acquired first edges 122. Here, for example, it is assumed that when the document information 120 is analyzed, one or more first edges 122 that are recognized as part of the diagram 101 and acquired include at least all first edges 122 that correspond to the edges 132 that form the diagram 101. On the other hand, for example, it is assumed that when the document information 120 is analyzed, one or more first edges 122 that are recognized as part of the diagram 101 and acquired may include first edges 122 that are not components that form the diagram 101.

[0034] The information processing device 100 identifies one or more edges 132 by extracting from one or more first edges 122 that will form the edge 132 of the diagram 101, in accordance with these assumptions. For example, the information processing device 100 identifies as edge 132 a first edge 122 that connects to a node identified by any of the labels 131 and to a node identified by any of the labels 131 from among the acquired one or more first edges 122.

[0035] As a result, the information processing device 100 can appropriately combine the results of analyzing the image information 110 and the results of analyzing the document information 120 to accurately identify one or more edges 132 that form the diagram 101. The information processing device 100 can accurately identify the structure of the diagram 101 using the identified labels 131, the nodes identified by the labels 131, and the identified edges 132.

[0036] (1-5) The information processing device 100 outputs the identified label 131, the node identified by the label 131, and the identified edge 132. The output format may be, for example, display on a screen, print to a printer, transmit to another computer, or store in a memory area. This makes the structure of the diagram 101 available to the information processing device 100. The information processing device 100 can, for example, make it easier to perform a predetermined task by referring to the structure of the diagram 101.

[0037] Here, we have described the case where the functions of the information processing device 100 are realized by a single computer, but this is not the only case. For example, the functions of the information processing device 100 may be realized through the collaboration of multiple computers. For example, the functions of the information processing device 100 may be realized on the cloud.

[0038] (An example of an information processing system 200) Next, an example of an information processing system 200 that applies the information processing device 100 shown in Figure 1 will be described using Figure 2.

[0039] Figure 2 is an explanatory diagram showing an example of an information processing system 200. In Figure 2, the information processing system 200 includes an information processing device 100, an image analysis device 201, a document analysis device 202, and a client device 203.

[0040] In the information processing system 200, the information processing device 100 and the image analysis device 201 are connected via a wired or wireless network 210. The network 210 can be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

[0041] Furthermore, in the information processing system 200, the information processing device 100 and the document analysis device 202 are connected via a wired or wireless network 210. Also, in the information processing system 200, the information processing device 100 and the client device 203 are connected via a wired or wireless network 210.

[0042] The information processing device 100 is a computer for identifying the structure of a diagram. The information processing device 100 receives a processing request that requests the identification of the structure of a diagram. The processing request may include, for example, image information of the diagram. The image information indicates an image in which the diagram is drawn. The processing request may also include, for example, document information of the diagram. The document information indicates a document in which the diagram is described.

[0043] A processing request may, for example, include one or more first labels that identify different nodes in a diagram, obtained by analyzing image information instead of image information. The nodes identified by the first labels may be spurious nodes that do not actually form a diagram. A processing request may, for example, include one or more second labels that identify different nodes in a diagram, obtained by analyzing document information instead of document information. The nodes identified by the second labels may be spurious nodes that do not actually form a diagram. A processing request may, for example, include one or more first edges in a diagram, obtained by analyzing document information instead of document information. The first edges may be spurious edges that do not actually form a diagram.

[0044] The processing request may further request execution of a predetermined task based on the structure of the diagram. The information processing apparatus 100 acquires the processing request by, for example, receiving the processing request from another computer. The information processing apparatus 100 may acquire the processing request by accepting an input of the processing request based on a user's operation input, for example.

[0045] The information processing apparatus 100 acquires image information and document information. The information processing apparatus 100 acquires the image information and the document information by, for example, extracting the image information and the document information from the processing request. The information processing apparatus 100 may acquire the image information and the document information by accepting inputs of the image information and the document information based on a user's operation input, for example.

[0046] The information processing apparatus 100 acquires one or more first labels that identify respective different nodes included in the diagram, which are obtained by analyzing the image information. The node identified by the first label may possibly be a false node that does not actually form a diagram. The information processing apparatus 100 transmits, to the image analysis apparatus 201, an image analysis request that includes the acquired image information and requests analysis of the image information, for example.

[0047] As a result of transmitting the image analysis request, the information processing apparatus 100 acquires the one or more first labels by receiving information indicating the one or more first labels from the image analysis apparatus 201. Further, when the one or more first labels can be extracted from the processing request, the information processing apparatus 100 does not need to transmit the image analysis request to the image analysis apparatus 201. In this case, the information processing system 200 may not include the image analysis apparatus 201, for example.

[0048] The information processing apparatus 100 acquires one or more second labels for identifying different nodes included in a diagram, and one or more first edges included in the diagram, which are obtained by analyzing document information. The node identified by the second label may possibly be a false node that does not actually form the diagram. The first edge may possibly be a false edge that does not actually form the diagram. The information processing apparatus 100 transmits, for example, a document analysis request that includes the acquired document information and requests analysis of the document information, to the document analysis apparatus 202.

[0049] As a result of transmitting the document analysis request, the information processing apparatus 100 acquires the one or more second labels and the one or more first edges by receiving information indicating the one or more second labels and the one or more first edges from the document analysis apparatus 202. Further, when the one or more second labels and the one or more first edges can be extracted from the processing request, the information processing apparatus 100 does not have to transmit the document analysis request to the document analysis apparatus 202. In this case, the information processing system 200 may not include the document analysis apparatus 202, for example.

[0050] Similar to FIG. 1, the information processing apparatus 100 identifies one or more labels for identifying different nodes forming the diagram, based on the result of associating the acquired one or more first labels with the acquired one or more second labels. Further, similar to FIG. 1, the information processing apparatus 100 identifies one or more edges forming the diagram, based on the acquired one or more first edges and the identified one or more labels.

[0051] The information processing apparatus 100 identifies the structure of the diagram based on the identified one or more labels and the identified one or more edges, and outputs structure information indicating the structure of the diagram. The information processing apparatus 100 transmits, for example, the structure information indicating the structure of the diagram to another computer. The other computer is, for example, the client apparatus 203 that is the transmission source of the processing request.

[0052] The information processing device 100 may output the result of performing a predetermined task based on the structure of the identified diagram. The information processing device 100 may, for example, send the result of performing the predetermined task to another computer. The other computer may be, for example, a client device 203 that sends the processing request. The information processing device 100 may be, for example, a server or a PC.

[0053] The image analysis device 201 is a computer for analyzing image information. The image analysis device 201 receives image analysis requests from other computers, such as the information processing device 100.

[0054] The image analysis device 201 analyzes image information in response to an image analysis request. By analyzing the image information, the image analysis device 201 identifies one or more first labels that identify different nodes included in the diagram. It is possible that the nodes identified by the first labels are false nodes that do not actually form part of the diagram.

[0055] The image analysis device 201 transmits information indicating one or more first labels that identify different nodes included in the diagram to another computer. The other computer is an information processing device 100 that is the source of the image analysis request. The image analysis device 201 is, for example, a server or a PC.

[0056] The document analysis device 202 is a computer for analyzing document information. The document analysis device 202 receives document analysis requests from other computers that request the analysis of document information. These other computers include the information processing device 100, etc.

[0057] The document analysis device 202 analyzes document information in response to a document analysis request. By analyzing the document information, the document analysis device 202 identifies one or more second labels that identify different nodes included in the diagram, and identifies one or more first edges included in the diagram. It is possible that the nodes identified by the second labels are false nodes that do not actually form part of the diagram. It is also possible that the first edges are false edges that do not actually form part of the diagram.

[0058] The document analysis device 202 transmits information to another computer indicating one or more second labels that identify different nodes included in the diagram, and one or more first edges included in the diagram. The other computer is an information processing device 100 that is the source of the document analysis request. The document analysis device 202 is, for example, a server or a PC.

[0059] The client device 203 is a computer used by a user who wishes to identify the structure of a diagram. Based on the user's input, the client device 203 sends a processing request to the information processing device 100 requesting that the structure of the diagram be identified.

[0060] The client device 203 receives structural information indicating the structure of the diagram from the information processing device 100. The client device 203 outputs structural information indicating the structure of the diagram. For example, the client device 203 outputs structural information indicating the structure of the diagram so that the user can refer to it.

[0061] The client device 203 may receive the results of performing a predetermined task from the information processing device 100. The client device 203 outputs the results of performing the predetermined task. For example, the client device 203 outputs the results of performing the predetermined task in a way that is accessible to the user. The client device 203 may be, for example, a PC, a tablet terminal, or a smartphone.

[0062] Here, we have described a case where the information processing device 100 is a different device from the image analysis device 201, but this is not limited to this case. For example, the information processing device 100 may have the functionality of an image analysis device 201 and may operate as an image analysis device 201. In this case, the information processing system 200 does not need to include an image analysis device 201.

[0063] Here, we have described a case where the information processing device 100 is a different device from the document analysis device 202, but this is not limited to this case. For example, the information processing device 100 may have the functionality of a document analysis device 202 and may operate as a document analysis device 202. In this case, the information processing system 200 does not need to include a document analysis device 202.

[0064] Here, we have described a case where the information processing device 100 is a different device from the client device 203, but this is not limited to this case. For example, the information processing device 100 may have the functionality of a client device 203 and may operate as a client device 203. In this case, the information processing system 200 does not need to include a client device 203.

[0065] This explanation describes a case where the image analysis device 201 is a different device from the document analysis device 202, but it is not limited to this case. For example, the image analysis device 201 may have the functionality of a document analysis device 202 and may operate as a document analysis device 202. In this case, the information processing system 200 does not need to include a document analysis device 202.

[0066] (Example of hardware configuration of the information processing device 100) Next, an example of the hardware configuration of the information processing device 100 will be described using Figure 3.

[0067] Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100. In Figure 3, the information processing device 100 includes a CPU (Central Processing Unit) 301, a memory 302, and a network interface 303. The information processing device 100 also includes a recording medium interface 304, a recording medium 305, a display 306, and an input device 307. Each component is connected by a bus 300.

[0068] Here, the CPU 301 is responsible for the overall control of the information processing device 100. The memory 302 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, flash ROM and ROM store various programs, and RAM is used as the work area for the CPU 301. Programs stored in memory 302 are loaded into the CPU 301, causing the CPU 301 to execute the coded processes.

[0069] The network interface 303 is connected to the network 210 via a communication line, and then connected to other computers via the network 210. The network interface 303 manages the internal interface with the network 210 and controls the input and output of data from other computers. The network interface 303 is, for example, a modem or a LAN adapter.

[0070] The recording medium interface 304 controls the reading and writing of data to the recording medium 305 according to the control of the CPU 301. The recording medium interface 304 is, for example, a disk drive, an SSD (Solid State Drive), or a USB (Universal Serial Bus) port. The recording medium 305 is a non-volatile memory that stores the data written under the control of the recording medium interface 304. The recording medium 305 is, for example, a disk, semiconductor memory, or USB memory. The recording medium 305 may be detachable from the information processing device 100.

[0071] Display 306 displays data such as cursors, icons, toolboxes, documents, images, or functional information. Display 306 is, for example, a CRT (Cathode Ray Tube), a liquid crystal display, or an organic EL (Electroluminescence) display. Input device 307 has keys for inputting characters, numbers, or various instructions, and performs data input. Input device 307 is, for example, a keyboard or a mouse. Input device 307 may also be, for example, a touch panel input pad or a numeric keypad.

[0072] The information processing device 100 may have, in addition to the components described above, a camera, for example. Furthermore, the information processing device 100 may have, in addition to the components described above, a printer, scanner, microphone, or speaker, for example. Also, the information processing device 100 may have, for example, multiple recording medium interfaces 304 and recording media 305. Furthermore, the information processing device 100 does not necessarily have, for example, a display 306 or an input device 307. Also, the information processing device 100 does not necessarily have, for example, recording medium interfaces 304 and recording media 305.

[0073] (Example of hardware configuration of image analysis device 201) The example of hardware configuration of the image analysis device 201 is the same as the example of hardware configuration of the information processing device 100 shown in Figure 3, so the explanation is omitted.

[0074] (Example of hardware configuration of document analysis device 202) The example of hardware configuration of the document analysis device 202 is the same as the example of hardware configuration of the information processing device 100 shown in Figure 3, so the explanation is omitted.

[0075] (Example of hardware configuration of client device 203) The example of hardware configuration of client device 203 is the same as the example of hardware configuration of information processing device 100 shown in Figure 3, so the explanation is omitted.

[0076] The following description primarily focuses on the case where the information processing device 100 operates independently. However, the content of the following description can also be applied to cases where the information processing device 100 collaborates with other computers, such as in the information processing system 200.

[0077] (An example of the functional configuration of the information processing device 100) Next, an example of the functional configuration of the information processing device 100 will be described using Figure 4.

[0078] Figure 4 is a block diagram showing an example of the functional configuration of the information processing device 100. In Figure 4, the information processing device 100 includes a storage unit 400, an acquisition unit 401, an image analysis unit 402, a document analysis unit 403, a correspondence relationship identification unit 404, a label identification unit 405, an edge identification unit 406, a structure identification unit 407, and an output unit 408.

[0079] The storage unit 400 is implemented by a storage area such as the memory 302 or recording medium 305 shown in Figure 3. The following description will focus on the case where the storage unit 400 is included in the information processing device 100, but is not limited to this case. For example, the storage unit 400 may be included in a device different from the information processing device 100, and the contents of the storage unit 400 may be accessible from the information processing device 100.

[0080] The acquisition unit 401 to the output unit 408 function as an example of a control unit. Specifically, the acquisition unit 401 to the output unit 408 realize their functions, for example, by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 or recording medium 305 shown in Figure 3, or by using the network interface 303. The processing results of each functional unit are stored in a storage area such as the memory 302 or recording medium 305 shown in Figure 3.

[0081] The storage unit 400 stores various types of information that are referenced or updated during the processing of each functional unit. The storage unit 400 stores the source data related to the diagram. The source data is the basis for the diagram's image information and document information. The image information indicates the image in which the diagram is drawn. The document information indicates the document in which the diagram is described. The source data is acquired, for example, by the acquisition unit 401. The source data may be pre-set, for example, by the user.

[0082] The storage unit 400 stores image information of the diagram. The image information, for example, represents an image of the diagram drawn at a predetermined resolution. The image information is generated, for example, by the image analysis unit 402. The image information may also be acquired, for example, by the acquisition unit 401. The image information may also be pre-set by the user, for example.

[0083] The storage unit 400 stores one or more first labels that identify different nodes included in the diagram, obtained by analyzing the image information. The first labels are, for example, strings. It is possible that the nodes identified by the first labels are false nodes that do not actually form a diagram. The first labels are acquired, for example, by the image analysis unit 402. The first labels may also be acquired, for example, by the acquisition unit 401. The first labels may be pre-set by the user, for example.

[0084] The storage unit 400 stores the document information of the diagram. The document information may be in XML format, draw.io format, or astah format, for example. The document information may be generated by the document analysis unit 403, for example. The document information may also be acquired by the acquisition unit 401, for example. The document information may be pre-set by the user, for example.

[0085] The storage unit 400 stores one or more second labels that identify different nodes included in the diagram, obtained by analyzing document information. The second labels are, for example, strings. It is possible that the nodes identified by the second labels are false nodes that do not actually form a diagram. The second labels are acquired, for example, by the document analysis unit 403. The second labels may also be acquired, for example, by the acquisition unit 401. The second labels may be pre-set by, for example, the user.

[0086] The storage unit 400 stores one or more first edges included in the diagram, obtained by analyzing document information. The first edges may be false edges that do not actually form part of the diagram. The first edges are acquired, for example, by the document analysis unit 403. The first edges may also be acquired, for example, by the acquisition unit 401. The first edges may be pre-set, for example, by the user.

[0087] The storage unit 400 stores correspondence information that associates one or more first labels with one or more second labels. The correspondence information indicates, for example, the association between any first label and any second label whose similarity is above a threshold. The correspondence information is generated, for example, by the correspondence relationship identification unit 404. The correspondence information may be acquired, for example, by the acquisition unit 401. The correspondence information may be pre-set by the user, for example.

[0088] The storage unit 400 stores structural information that indicates the structure of the diagram. The structural information includes, for example, one or more labels that identify each different node that forms the diagram. The structural information may, for example, indicate each different node that forms the diagram. The structural information may, for example, indicate one or more edges that form the diagram. The structural information is generated, for example, by the structure identification unit 407.

[0089] The acquisition unit 401 acquires various types of information used in the processing of each functional unit. The acquisition unit 401 stores the acquired information in the storage unit 400 or outputs it to each functional unit. The acquisition unit 401 may also output the information stored in the storage unit 400 to each functional unit. The acquisition unit 401 acquires various types of information, for example, based on user input. The acquisition unit 401 may also receive various types of information from a device other than the information processing device 100, for example.

[0090] The acquisition unit 401 acquires, for example, a processing request that requests the identification of the structure of a diagram. The processing request may include, for example, raw data relating to the diagram. The processing request may not include, for example, raw data but include image information of the diagram. The processing request may not include, for example, raw data but include document information of the diagram.

[0091] A processing request may include, for example, one or more first labels that identify different nodes in a diagram, obtained by analyzing image information. In this case, the processing request does not need to include image information.

[0092] A processing request may include, for example, one or more second labels that identify different nodes in a diagram, obtained by analyzing document information, and one or more first edges in the diagram. In this case, the processing request does not have to include document information. The processing request may also include correspondence information that associates one or more first labels with one or more second labels.

[0093] Specifically, the acquisition unit 401 acquires processing requests by receiving input processing requests. Specifically, the acquisition unit 401 may acquire processing requests by receiving processing requests from other computers. Other computers are, for example, client devices 203.

[0094] The acquisition unit 401 acquires, for example, raw data. The acquisition unit 401 acquires raw data by extracting it from a processing request. Specifically, the acquisition unit 401 may acquire raw data by accepting raw data as input. Specifically, the acquisition unit 401 may acquire raw data by receiving raw data from another computer. The other computer is, for example, a client device 203.

[0095] The acquisition unit 401 acquires image information, for example. The acquisition unit 401 acquires image information by extracting it from a processing request. Specifically, the acquisition unit 401 may acquire image information by accepting image information as input. Specifically, the acquisition unit 401 may acquire image information by receiving image information from another computer. The other computer is, for example, a client device 203.

[0096] The acquisition unit 401 acquires document information, for example. The acquisition unit 401 acquires document information by extracting it from a processing request. Specifically, the acquisition unit 401 may acquire document information by accepting input of document information. Specifically, the acquisition unit 401 may acquire document information by receiving document information from another computer. The other computer is, for example, a client device 203.

[0097] The acquisition unit 401 acquires, for example, one or more first labels. The acquisition unit 401 acquires one or more first labels by extracting one or more first labels from a processing request. Specifically, the acquisition unit 401 may acquire one or more first labels by accepting one or more first labels as input. Specifically, the acquisition unit 401 may acquire one or more first labels by receiving one or more first labels from another computer. The other computer is, for example, a client device 203.

[0098] The acquisition unit 401 acquires, for example, one or more second labels. The acquisition unit 401 acquires one or more second labels by extracting one or more second labels from a processing request. Specifically, the acquisition unit 401 may acquire one or more second labels by accepting one or more second labels as input. Specifically, the acquisition unit 401 may acquire one or more second labels by receiving one or more second labels from another computer. The other computer is, for example, a client device 203.

[0099] The acquisition unit 401 acquires, for example, one or more first edges. The acquisition unit 401 acquires one or more first edges by extracting one or more first edges from a processing request. Specifically, the acquisition unit 401 may acquire one or more first edges by receiving one or more first edges as input. Specifically, the acquisition unit 401 may acquire one or more first edges by receiving one or more first edges from another computer. The other computer is, for example, a client device 203.

[0100] The acquisition unit 401 acquires, for example, correspondence information. The acquisition unit 401 acquires correspondence information by extracting it from a processing request. Specifically, the acquisition unit 401 may acquire correspondence information by accepting the input of correspondence information. Specifically, the acquisition unit 401 may acquire correspondence information by receiving correspondence information from another computer. The other computer is, for example, a client device 203.

[0101] The acquisition unit 401 may receive a start trigger to initiate processing in any of the functional units. A start trigger could be, for example, a predetermined operation input by a user. Another start trigger could be, for example, the reception of predetermined information from another computer. Yet another start trigger could be, for example, the output of predetermined information by any of the functional units. The acquisition unit 401 may also accept the acquisition of a processing request as a start trigger to initiate processing in the image analysis unit 402, the document analysis unit 403, the correspondence relationship identification unit 404, the label identification unit 405, the edge identification unit 406, and the structure identification unit 407.

[0102] The image analysis unit 402 acquires image information by generating image information based on the raw data acquired by the acquisition unit 401. The image analysis unit 402 generates image information, for example, when the acquisition unit 401 does not acquire image information. Specifically, the image analysis unit 402 generates image information that shows an image of a diagram drawn at a predetermined resolution. This allows the image analysis unit 402 to prepare image information to be analyzed.

[0103] The image analysis unit 402 obtains one or more first labels that identify different nodes included in the diagram by analyzing the image information acquired by the acquisition unit 401 or the image analysis unit 402. The image analysis unit 402 may obtain one or more first labels by analyzing the image information, for example, using a large-scale language model. The image analysis unit 402 may also obtain one or more first labels by analyzing the image information, for example, using a dedicated detector for analyzing image information. As a result, the image analysis unit 402 can identify nodes that have a relatively high likelihood of being components forming the diagram through one or more first labels.

[0104] The image analysis unit 402 may divide the image based on the image information such that there are fewer diagram components drawn overlapping at the boundaries dividing the image on which the diagram is drawn. Based on the image information, the image analysis unit 402 obtains one or more first labels by analyzing each divided part of the image. As a result, the image analysis unit 402 can identify nodes that have a relatively high probability of being components forming the diagram via one or more first labels. The image analysis unit 402 can accurately identify one or more first labels even if the image size is large.

[0105] The document analysis unit 403 acquires document information by generating document information based on the raw data acquired by the acquisition unit 401. The document analysis unit 403 generates document information, for example, when the acquisition unit 401 does not acquire document information. Specifically, the document analysis unit 403 generates document information by extracting the parts of the raw data that are recognized as document information using a rule-based approach. This allows the document analysis unit 403 to prepare the document information to be analyzed.

[0106] The document analysis unit 403 analyzes the document information acquired by the acquisition unit 401 or the document analysis unit 403 to obtain one or more second labels that identify different nodes included in the diagram, and one or more first edges included in the diagram. The document analysis unit 403 may, for example, use a rule-based method to analyze the document information to obtain one or more second labels and one or more first edges. The document analysis unit 403 may, for example, use a large-scale language model to analyze the document information to obtain one or more second labels and one or more first edges. This allows the document analysis unit 403 to identify first edges that have a relatively high likelihood as components forming the diagram. The document analysis unit 403 can identify second labels that have a relatively high likelihood as strings.

[0107] The correspondence identification unit 404 associates one or more first labels acquired by the acquisition unit 401 or the image analysis unit 402 with one or more second labels acquired by the acquisition unit 401 or the document analysis unit 403. For example, the correspondence identification unit 404 associates at least one first label of any of the one or more first labels with one or more second labels of any of the one or more second labels.

[0108] Specifically, the correspondence identification unit 404 calculates the similarity between each of the one or more first labels and each of the one or more second labels. The similarity to be calculated is preferably string-like similarity rather than semantic similarity. The similarity is, for example, the Levenshtein edit distance. The similarity may also be, for example, cosine similarity.

[0109] Specifically, the correspondence identification unit 404 associates any first label whose similarity is above a threshold with any second label. The threshold is set in advance by the user, for example. As a result, the information processing device 100 can associate the results of analyzing the image information with the results of analyzing the document information and obtain guidelines to identify one or more labels that identify different nodes forming the diagram.

[0110] The label identification unit 405 identifies one or more labels that identify different nodes forming the diagram, based on the results of the correspondence identification unit 404. The label identification unit 405 identifies one or more labels by correcting one or more first labels obtained based on the correspondence results.

[0111] Specifically, the label identification unit 405 corrects one or more first labels that identify distinct nodes that are likely to be components of the diagram, based on one or more second labels that are likely to be strings. Here, specifically, the label identification unit 405 corrects any of the first labels based on any of the second labels that are associated with that particular first label from among the one or more second labels. Specifically, the label identification unit 405 identifies the corrected one or more first labels as one or more labels that identify distinct nodes that form the diagram.

[0112] More specifically, the label identification unit 405 determines whether any of the first labels match any of the second labels associated with that first label. More specifically, if the label identification unit 405 determines that they do not match, it corrects the first label to match the most likely second label, and identifies the corrected first label as the label that identifies the node forming the diagram. More specifically, if the label identification unit 405 determines that they do match, it identifies the first label as is as the label that identifies the node forming the diagram.

[0113] As a result, the label identification unit 405 can accurately identify one or more labels that identify different nodes forming the diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information. The label identification unit 405 can obtain a guideline for identifying one or more edges forming the diagram.

[0114] The label identification unit 405 may, for example, identify one or more labels by removing any of the acquired second labels that cannot be associated with any of the acquired first labels, based on the matching results.

[0115] Specifically, the label identification unit 405 removes from the list of one or more second labels any second label that is unlikely to be a string and does not correspond to any of the first labels that identify nodes that are likely to be components of the diagram. Specifically, the label identification unit 405 identifies the one or more second labels that remain after the removal, each corresponding to a different node that is likely to be a component of the diagram, as one or more labels.

[0116] As a result, the label identification unit 405 can accurately identify one or more labels that identify different nodes forming the diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information. The label identification unit 405 can obtain a guideline for identifying one or more edges forming the diagram.

[0117] The edge identification unit 406 identifies one or more edges that form a diagram from among one or more first edges based on one or more labels identified by the label identification unit 405. The edge identification unit 406 identifies, for example, one of the first edges that is connected to a node identified by any of the one or more identified labels and to a node identified by any of the one or more identified labels. The edge identification unit 406 may further identify, for example, one of the first edges that has one end connected to a node identified by any of the one or more identified labels and the other end not connected to any node as an edge that forms a diagram.

[0118] As a result, the edge identification unit 406 can accurately identify one or more edges that form a diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information. For example, the edge identification unit 406 can identify a first edge whose both ends are connected to nodes that are likely to be components of the diagram as an edge that is likely to be a component of the diagram. For example, the edge identification unit 406 can determine that a first edge whose at least one end is not connected to a node that is likely to be a component of the diagram is unlikely to be a component of the diagram.

[0119] The structure identification unit 407 generates structural information indicating the structure of the diagram based on one or more labels identified by the label identification unit 405 and one or more edges identified by the edge identification unit 406. For example, the structure identification unit 407 generates structural information indicating the structure of the diagram formed by one or more labels, the nodes identified by each of the one or more labels, and one or more edges. This allows the structure identification unit 407 to accurately identify the structure of the diagram.

[0120] The structure identification unit 407 may, based on the results of the correspondence identification unit 404, identify any of the one or more second labels that cannot be associated with any of the one or more first labels. The structure identification unit 407 associates the identified second label with any of the one or more edges whose distance on the image from the identified second label is less than or equal to a threshold. The structure identification unit 407 generates further structural information by associating the identified second label with the identified edge. This allows the structure identification unit 407 to add information that supplements the meaning of the edges forming the diagram to the structural information.

[0121] The output unit 408 outputs the processing result of at least one of the functional units. The output format can be, for example, display on a screen, print to a printer, transmit to an external device via the network interface 303, or store in a storage area such as the memory 302 or recording medium 305. This allows the output unit 408 to notify the user of the processing result of at least one of the functional units, thereby improving the usability of the information processing device 100.

[0122] The output unit 408 outputs, for example, the structural information generated by the structural identification unit 407. Specifically, the output unit 408 outputs the structural information so that it can be viewed by the user. Specifically, the output unit 408 transmits the structural information to another computer. The other computer is, for example, the client device 203. This allows the output unit 408 to make the structure of the diagram available externally.

[0123] Here, we have described a case in which the information processing device 100 includes an acquisition unit 401, an image analysis unit 402, a document analysis unit 403, a correspondence relationship identification unit 404, a label identification unit 405, an edge identification unit 406, a structure identification unit 407, and an output unit 408, but it is not limited to this. For example, the information processing device 100 may not include an image analysis unit 402. In this case, the information processing device 100 specifically cooperates with an image analysis device 201 which becomes the image analysis unit 402. For example, the information processing device 100 may not include a document analysis unit 403. In this case, the information processing device 100 specifically cooperates with a document analysis device 202 which becomes the document analysis unit 403.

[0124] (Specific example of the functional configuration of the information processing device 100) Next, a specific example of the functional configuration of the information processing device 100 will be explained using Figure 5.

[0125] Figure 5 is a block diagram showing a specific example of the functional configuration of the information processing device 100. The information processing device 100 can utilize an image-compatible large-scale language model 500. The information processing device, for example, has the large-scale language model 500 executable on its own device. The information processing device 100 includes a text input unit 501.

[0126] The information processing device 100 includes an image conversion unit 502, an image splitting unit 503, a node recognition unit 504, and an edge recognition unit 505. The information processing device 100 includes a document extraction unit 506, a node recognition unit 507, and an edge recognition unit 508. The information processing device 100 includes a mapping processing unit 509, a filtering unit 510, and a label correction unit 511. The information processing device 100 includes an edge correction unit 512 and a semantic correction unit 513.

[0127] The text input unit 501 accepts input of image information of the diagram and source data that will form the basis of document information of the diagram, based on user input. The image information indicates the image in which the diagram is drawn. The document information indicates the document in which the diagram is described.

[0128] The image conversion unit 502 generates image information of the diagram. The image conversion unit 502 generates image information by, for example, converting the original data into image information. The image conversion unit 502 may also convert the document information obtained by converting the original data into image information. The document information is generated by the document extraction unit 506. The image splitting unit 503 splits the image indicated by the image information. The image splitting unit 503 splits the image, for example, if the size of the image is greater than or equal to a threshold. The image splitting unit 503 preferably splits the image in such a way that there are fewer diagram components drawn overlapping at the boundaries where the image is split.

[0129] The node recognition unit 504 analyzes the image indicated by the image information using a large-scale language model. The node recognition unit 504 analyzes the image indicated by the image information for each divided part. Based on the results of the image analysis, the node recognition unit 504 recognizes the first node included in the diagram. Based on the results of the image analysis, the node recognition unit 504 recognizes a first label that identifies the recognized first node.

[0130] The edge recognition unit 505 analyzes the image indicated by the image information using a large-scale language model. The edge recognition unit 505 analyzes the image indicated by the image information for each divided part. Based on the results of the image analysis, the edge recognition unit 505 recognizes a first edge that is included in the diagram and whose ends are connected to the first node.

[0131] The document extraction unit 506 generates document information for the diagram. The document extraction unit 506 generates document information, for example, by converting the original data into document information.

[0132] The node recognition unit 507 analyzes the document information indicated by the document information in a rule-based manner. Based on the results of the document analysis, the node recognition unit 507 recognizes the second node included in the diagram. Based on the results of the document analysis, the node recognition unit 507 recognizes a second label that identifies the recognized second node.

[0133] The edge recognition unit 508 analyzes the document indicated by the document information using a large-scale language model. The node recognition unit 507 analyzes the document indicated by the document information for each divided part. Based on the results of the document analysis, the edge recognition unit 508 recognizes a second edge included in the diagram, where both ends are connected to the second node.

[0134] Here, the node recognition unit 504 has the advantage of being able to recognize the first node with higher accuracy than the node recognition unit 507 can recognize the second node. However, due to factors such as image resolution, the node recognition unit 504 may be able to recognize the first label with lower accuracy than the node recognition unit 507 can recognize the second label. Also, due to factors such as the complexity of the diagram, the edge recognition unit 505 may be able to recognize the first edge with lower accuracy than the edge recognition unit 508 can recognize the second edge.

[0135] On the other hand, due to noise such as strings representing nodes other than those present in the document, the accuracy with which the node recognition unit 507 recognizes the second node may be lower than the accuracy with which the node recognition unit 504 recognizes the first node. Furthermore, the accuracy with which the node recognition unit 507 recognizes the second label is considered to be higher than the accuracy with which the node recognition unit 504 recognizes the first label. Also, the accuracy with which the edge recognition unit 508 recognizes the second edge is considered to be higher than the accuracy with which the edge recognition unit 505 recognizes the first edge. For example, although the edge recognition unit 508 may mistakenly recognize false edges that do not form a diagram, it tends to comprehensively recognize edges that at least form a diagram.

[0136] Therefore, the information processing device 100 accurately recognizes the structure of the diagram by comparing the first node, first label, and first edge recognized by analyzing the image with the second node, second label, and second edge recognized by analyzing the document.

[0137] The mapping processing unit 509 associates each first node with each second node. The mapping processing unit 509 calculates, for example, the similarity between each first label and each second label. The mapping processing unit 509 associates, for example, any first label with any second label whose similarity is above a threshold.

[0138] As a result, the mapping processing unit 509 can associate the first node, which is recognized with relatively high accuracy, with the second label, which is also recognized with relatively high accuracy, via the first label. Since the first node, which is recognized with relatively high accuracy, has been associated with the first label via the second label, the mapping processing unit 509 can identify the second node, which is considered to have a relatively high probability of being a component of the diagram.

[0139] If the filtering unit 510 finds any second node that cannot be associated with any of the first nodes by the mapping processing unit 509, it removes that second node from the set of second nodes recognized by the node recognition unit 507. As a result, the filtering unit 510 can retain second nodes that have been associated with the first nodes that were recognized with relatively high accuracy and are therefore considered to have a relatively high probability of being components of the diagram.

[0140] The label correction unit 511 sets labels that identify the nodes forming the diagram based on the first label and the second label. The label correction unit 511 sets the corrected first label as a label forming the diagram by correcting each first label to match the second label associated with that first label. The label correction unit 511 sets the first node identified by the corrected first label as a node forming the diagram. In this way, the label correction unit 511 can set appropriate nodes forming the diagram and labels that identify those nodes.

[0141] The edge correction unit 512 sets the edges that form the diagram based on the set of second nodes left by the filtering unit 510. The edge correction unit 512 sets, for example, second edges that are connected to different second nodes in the set at both ends, or to the same second node in the set at both ends, as edges that form the diagram. In this way, the edge correction unit 512 can extract appropriate edges that actually form the diagram from the set of second edges recognized by the edge recognition unit 508, which comprehensively includes at least the edges that form the diagram.

[0142] The semantic correction unit 513 may associate semantic information with edges. For example, if there is a second label that cannot be associated with any of the first labels by the association processing unit 509, the semantic correction unit 513 identifies that second label. The semantic correction unit 513 associates the identified second label as semantic information with an edge that forms a diagram, which corresponds to a second edge whose distance on the image from that second label is below a threshold. In this way, the semantic correction unit 513 can assign semantic information to edges so that the meaning of the edges is easier to understand.

[0143] The information processing device 100 executes task processing 514 based on the nodes that form the set diagram, labels that identify the nodes, the edges that form the set diagram, and the semantic information associated with the edges. This allows the information processing device 100 to execute task processing 514 with high accuracy.

[0144] (Operation Flow of the Information Processing Device 100) Next, the operation flow of the information processing device 100 will be explained using Figures 6 to 12. First, an example of a diagram that the information processing device 100 recognizes will be explained using Figures 6 and 7.

[0145] Figures 6 and 7 are explanatory diagrams showing an example of a diagram. In Figure 6, the diagram is, for example, a screen system diagram 600. The screen system diagram 600 includes a node 601 and an edge 602. The screen system diagram 600 includes components 610 that may be noise but are not node 601. Node 601 has a label such as "FD001F01: Login".

[0146] For example, conventionally, it may not be possible to accurately recognize the label of node 601. Also, conventionally, it may be possible to mistakenly recognize component 610 as node 601. Also, conventionally, it may be possible to mistakenly recognize the connection destination of edge 602 as component 610. Also, conventionally, it may be impossible to correctly recognize the branch of edge 602. Therefore, it is sometimes desirable to accurately recognize node 601, the label that identifies node 601, and edge 602. Next, we will move on to the explanation of Figure 7.

[0147] In Figure 7, the diagram is, for example, a sequence diagram 700. The sequence diagram 700 includes a node 701 and an edge 702. The sequence diagram 700 includes components 710, 711, and 720 that may be noise but are not node 701. Node 701 has a label such as, for example, "FD104: Store Information Confirmation (Store) (SH-02)". Here, we have described a case where component 720 is not treated as node 701, but this is not limited to that. For example, component 720 may be treated as node 701.

[0148] For example, conventional systems may not be able to accurately recognize the label of node 701. Also, conventional systems may mistakenly recognize components 710, 711, and 720 as node 701. Furthermore, conventional systems may mistakenly recognize the connection destination of edge 702 as components 710, 711, and 720. Also, conventional systems may not be able to correctly recognize the branching of edge 702. Therefore, it is desirable to be able to accurately recognize node 701, the label that identifies node 701, and edge 702.

[0149] Next, using Figures 8 to 11, we will explain an example of the source data and document information acquired by the information processing device 100.

[0150] Figures 8 to 11 are explanatory diagrams showing an example of raw data and document information. In Figure 8, the information processing device 100 acquires raw data 800 in XML format. The raw data 800 represents the drawing position of text boxes and the text within the text boxes. Text boxes may correspond to nodes, for example. Next, we will move on to the explanation of Figure 9.

[0151] In Figure 9, the information processing device 100 generates document information 900 suitable for analysis by extracting diagram-related information from the original data 800 using a rule-based approach. The information processing device 100 may also generate image information based on the original data 800 or the document information 900. The image information may, for example, show an image of the screen system diagram 600 shown in Figure 6. Next, we will move on to the explanation of Figure 10.

[0152] As shown in Figure 10, the information processing device 100 may acquire the original data 1000 in draw.io format instead of the original data 800 in XML format. <mxCell ... / > defines a Cell that will be a component of the diagram. The original data 1000 represents the drawing position of the Cell and the text within the Cell. A Cell may correspond to a node, for example. Next, we will move on to the explanation of Figure 11.

[0153] In Figure 11, the information processing device 100 generates document information 1100 suitable for analysis by extracting information related to the diagram from the original data 1000 in a rule-based manner. The information processing device 100 may also generate image information based on the original data 1000 or the document information 1100. The image information may, for example, show an image of the screen system diagram 600 shown in Figure 6.

[0154] Next, the operation flow of the information processing device 100 will be explained using Figure 12.

[0155] Figure 12 is an explanatory diagram showing the operation flow of the information processing device 100. In Figure 12, the information processing device 100 is assumed to have acquired image information 1210 and document information 1220, similar to Figures 8 to 11. Based on the image information 1210 and document information 1220, the information processing device 100 identifies the structure of the diagram.

[0156] As described above, when nodes are identified by analyzing image information 1210, it is thought that the accuracy of identifying the nodes forming the diagram is higher compared to when nodes are identified by analyzing document information 1220. On the other hand, when edges are identified by analyzing document information 1220, it is thought that the accuracy of identifying the edges forming the diagram is higher compared to when edges are identified by analyzing image information 1210.

[0157] Furthermore, when identifying labels for identifying nodes by analyzing document information 1220, it is considered that the string-like accuracy of the labels is higher compared to when identifying labels for identifying nodes by analyzing image information 1210. Therefore, the information processing device 100 takes the above-mentioned features into consideration and implements a hybrid method for identifying the structure of a diagram that incorporates the advantages of analyzing image information 1210 and the advantages of analyzing document information 1220. Specific examples of this hybrid method will be described later in Figures 13 to 16.

[0158] As a result, the information processing device 100 can accurately identify nodes that form a diagram and labels that identify those nodes by appropriately combining the results of analyzing the image information 1210 and the results of analyzing the document information 1220. The information processing device 100 can accurately identify edges that form a diagram by appropriately combining the results of analyzing the image information 1210 and the results of analyzing the document information 1220. Based on the identified nodes, labels that identify those nodes, and identified edges, the information processing device 100 can accurately identify the structure of the diagram. The information processing device 100 can accurately perform predetermined tasks that refer to the structure of the diagram.

[0159] (An example of the operation of the information processing device 100) Next, an example of the operation of the information processing device 100 will be explained using Figures 13 to 15. First, using Figure 13, an example of the information processing device 100 acquiring image information and document information will be explained.

[0160] Figure 13 is an explanatory diagram illustrating an example of acquiring image information and document information. In Figure 13, the information processing device 100 acquires raw data in XML format that defines a screen 1300 explaining a diagram. The information processing device 100 analyzes the acquired raw data, extracts information related to the diagram from the raw data using a rule-based approach, and acquires document information by generating document information that includes the extracted information.

[0161] Furthermore, the information processing device 100 acquires image information by extracting an image 1310 on which a diagram is drawn from the screen 1300 based on the user's input, and generating image information representing the image 1310. In this case, it is preferable that the information processing device 100 generates image information representing the image 1310 at a predetermined resolution or higher in order to enable accurate recognition of the string. For example, it is preferable that the string on the image 1310 is 12 pixels or larger.

[0162] Next, an example of the operation of the information processing device 100 will be described using Figures 14 and 15.

[0163] Figures 14 and 15 are explanatory diagrams showing an example of the operation of the information processing device 100. In Figure 14, the information processing device 100 uses a large-scale language model to analyze the image 1310 shown by the image information to identify a first node that is a candidate for forming a diagram and a first label that identifies the first node. The information processing device 100 may further identify a first edge that is a candidate for forming a diagram by analyzing the image 1310 shown by the image information using a large-scale language model.

[0164] Here, in a large-scale language model, it is conceivable that there may be input constraints on image information. The input constraint may be, for example, an upper limit on the amount of data in the image information. The input constraint may also be, for example, an upper limit on the overall size, height, or width of the image 1310 represented by the image information. For example, the input constraint may be that image 1310 fits within 2048 pixels vertically and 2048 pixels horizontally.

[0165] In this case, it is preferable for the information processing device 100 to use a large-scale language model to divide the image 1310 before analysis. However, depending on the boundary where the image 1310 is divided, the accuracy of analyzing the image 1310 may deteriorate. For example, if the drawn nodes are divided at the boundary where the image 1310 is divided, the accuracy of identifying those nodes and the labels that identify those nodes is likely to deteriorate.

[0166] Therefore, the information processing device 100 divides the image 1310 in such a way that the number of diagram components drawn overlapping at the boundaries dividing the image 1310 is reduced. The information processing device 100, for example, binarizes the image 1310 into black and white. The information processing device 100, for example, searches for a boundary to divide the image 1310 based on the binarized image 1310, in the vertical and horizontal directions, such that the number of black pixels present on the boundary is minimized, according to the input constraints. The information processing device 100, for example, divides the image 1310 at the discovered boundary.

[0167] In the example shown in Figure 14, the information processing device 100 specifically divides image 1310 into images 1401 and 1402. Specifically, the information processing device 100 uses a large-scale language model to analyze each divided portion of image 1310, thereby identifying a first node that is a candidate for forming a diagram and a first label that identifies the first node, as shown in the recognition result table 1410. Each block in the recognition result table 1410 corresponds to the first node. The text in each block of the recognition result table 1410 corresponds to the first label that identifies the first node.

[0168] Furthermore, the information processing device 100 analyzes the document indicated by the document information to identify a second node that is a candidate for forming a diagram and a second label that identifies the second node, as shown in the recognition result table 1420. Each block in the recognition result table 1420 corresponds to a second node. The text in each block in the recognition result table 1420 corresponds to a second label that identifies the second node. In addition, the information processing device 100 analyzes the document indicated by the document information to identify a second edge that is a candidate for forming a diagram, as shown in the recognition result table 1520 described later in Figure 15.

[0169] The information processing device 100 associates the first label with the second label based on the recognition result tables 1410 and 1420. The information processing device 100 calculates, for example, the similarity between each first label and each second label. The information processing device 100 associates, for example, any first label with a second label whose similarity is above a threshold.

[0170] If there is a second label that does not correspond to any of the first labels, the information processing device 100 removes the second node identified by that second label from the set of identified second nodes. In the example in Figure 14, specifically, the information processing device 100 removes the second node identified by the second label "modal call" from the set of second nodes.

[0171] This allows the information processing device 100 to identify a second node, which is a false node that does not form a node in the diagram. The information processing device 100 can identify a second label, which is a string that does not serve as a label to identify a node. For example, the information processing device 100 can identify a second label, "modal call," which is a string that does not serve as a label to identify a node.

[0172] The information processing device 100 determines whether a first label matches a second label associated with that first label. If the information processing device 100 determines that any first label does not match any second label associated with that first label, it corrects the first label to match the second label. In the example in Figure 14, the information processing device 100 corrects the first label "FD101: Store Information Confirmation (Company)" to the first label "FD101: Store Information Confirmation (Head Office)" to match the second label "FD101: Store Information Confirmation (Head Office)".

[0173] As a result, the information processing device 100 can appropriately correct the first label to match the second label, which has relatively high string-like precision. Therefore, the information processing device 100 can accurately identify the labels that identify the nodes forming the diagram in a string-like manner. In this way, the information processing device 100 can correct both the combination of the first node and the first label, and the combination of the second node and the second label, so that they both represent the combination of nodes and labels forming the diagram. Next, we will move on to the explanation of Figure 15.

[0174] In Figure 15, the information processing device 100 identifies the combinations of nodes and labels that form the diagram, represented by the combination of the first node and the first label, and the combination of the second node and the second label, as shown in the recognition result table 1510. Each block in the recognition result table 1510 corresponds to an edge. The text in each block of the recognition result table 1510 corresponds to a combination of a label that identifies the node from which the edge is connected and a label that identifies the node to which the edge is connected. Furthermore, as shown in the recognition result table 1520, the information processing device 100 has already identified a second edge that is a candidate for forming the diagram.

[0175] The set of second edges is expected to comprehensively include at least the true second edges that form a diagram, although it may mistakenly include false second edges that do not form a diagram. Based on the recognition result tables 1510 and 1520, the information processing device 100 extracts the true second edges that form a diagram from the set of second edges.

[0176] Here, for example, if the second edge is a true edge, then it is assumed that both ends of the second edge are not connected to any second nodes that have been removed from the set of second nodes and determined to be false nodes. In other words, if the second edge is a true edge, then it is assumed that both ends of the second edge are connected to any of the second nodes remaining in the set of second nodes. Therefore, the information processing device 100 identifies a second edge whose ends are connected to any of the second nodes in the set of second nodes as an edge that forms a diagram.

[0177] In the example shown in Figure 15, the information processing device 100 identifies the second edge "FD001F01: Login → FD002F01: Menu" as an edge that forms the diagram. The information processing device 100 determines the second edge "Screen ID: Screen Name → Modal Call" as a false edge.

[0178] As a result, the information processing device 100 can accurately identify the nodes that form the diagram, the labels that identify those nodes, and the edges that form the diagram, and can accurately identify the structure of the diagram. Based on the structure of the diagram, the information processing device 100 can perform predetermined tasks.

[0179] Here, the information processing device 100 may further associate semantic information explaining the meaning of the edges that form the diagram. Next, an example of how the information processing device 100 associates semantic information with edges will be explained using Figure 16.

[0180] Figure 16 is an explanatory diagram illustrating an example of associating semantic information with edges. In Figure 16, the diagram is a sequence diagram 1600. Here, the information processing device 100 has already identified a second node that is a candidate for forming the diagram and a second label that identifies the said second node, as shown in the recognition result table 1610.

[0181] The information processing device 100 identifies the nodes that form the diagram, the labels that identify those nodes, and the edges that form the diagram, similar to Figures 13 to 15. At this time, the information processing device 100 identifies the second labels corresponding to the second nodes that were removed from the identified set of second nodes and were determined to be false nodes, as candidates for semantic information to be assigned to the edges.

[0182] The information processing device 100 determines whether or not there is a second edge whose distance on the image to the second node corresponding to the identified second label is below a threshold. If a second edge exists, the information processing device 100 associates the identified second label with the edge corresponding to that second edge as semantic information to be attached to the edge forming the diagram. If a second edge does not exist, the information processing device 100 discards the identified second label. In this way, the information processing device 100 can attach semantic information to the edges, making it easier for the user to understand the structure of the diagram.

[0183] (Overall Processing Procedure) Next, an example of the overall processing procedure executed by the information processing device 100 will be described using Figure 17. The overall processing is realized, for example, by the CPU 301 shown in Figure 3, storage areas such as memory 302 and recording medium 305, and network I / F 303.

[0184] Figure 17 is a flowchart showing an example of the overall processing procedure. In Figure 17, the information processing device 100 acquires document information (step S1701).

[0185] Next, the information processing device 100 generates an image of the diagram based on the document information (step S1702). Then, the information processing device 100 divides the image of the diagram (step S1703).

[0186] Next, the information processing device 100 identifies one or more first nodes by analyzing each divided portion of the diagram image using a large-scale language model (step S1704). Then, the information processing device 100 identifies one or more first edges by analyzing each divided portion of the diagram image using a large-scale language model (step S1705).

[0187] Furthermore, the information processing device 100 extracts documents related to the diagram based on document information using a rule-based approach (step S1706). Next, the information processing device 100 identifies one or more second nodes by analyzing the extracted documents (step S1707). Then, the information processing device 100 identifies one or more second edges by analyzing the extracted documents (step S1708).

[0188] Next, the information processing device 100 associates one or more first nodes with one or more second nodes (step S1709). Then, based on the association results, the information processing device 100 identifies the nodes that form the diagram (step S1710).

[0189] Next, the information processing device 100 corrects the labels that identify the nodes forming the diagram based on the matching results (step S1711). Then, the information processing device 100 identifies the edges forming the diagram based on the matching results (step S1712).

[0190] Next, the information processing device 100 sets semantic information for the edges forming the diagram based on the matching results (step S1713). Then, the information processing device 100 performs a predetermined task based on the structure of the diagram including the identified nodes and identified edges (step S1714). After that, the information processing device 100 completes the entire process.

[0191] Here, the information processing device 100 may execute some of the steps in the flowchart of Figure 17 in a different order. For example, the order of steps S1702 to S1705 and steps S1706 to S1708 can be swapped. Also, the information processing device 100 may omit some of the steps in the flowchart of Figure 17. For example, steps S1713 and S1714 can be omitted.

[0192] As described above, the information processing device 100 can obtain one or more first labels that identify different nodes included in the diagram, obtained by analyzing the image information of the diagram. The information processing device 100 can obtain one or more second labels that identify different nodes included in the diagram, obtained by analyzing the document information of the diagram. The information processing device 100 can obtain one or more first edges included in the diagram, obtained by analyzing the document information of the diagram. The information processing device 100 can associate one or more first labels obtained from the one or more first labels with one or more second labels obtained from the one or more second labels. Based on the association result, the information processing device 100 can identify one or more labels that identify the nodes forming the diagram. Based on the identified one or more labels, the information processing device 100 can identify one or more edges that form the diagram from among the one or more first edges obtained. As a result, the information processing device 100 can accurately identify the structure of the diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information.

[0193] According to the information processing device 100, a first label of any one or more acquired first labels whose similarity is above a threshold can be associated with a second label of any one or more acquired second labels. According to the information processing device 100, one or more labels can be identified by correcting the one or more acquired first labels based on the association result. In this way, the information processing device 100 can accurately identify one or more labels that identify different nodes forming a diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information.

[0194] According to the information processing device 100, with respect to any of the first labels, the first label can be corrected based on any of the second labels associated with the first label from among the one or more second labels that have been acquired. As a result, the information processing device 100 can appropriately correct the first label and accurately identify the labels that identify the nodes forming the diagram.

[0195] According to the information processing device 100, from the acquired one or more first edges, it is possible to identify a node identified by any of the one or more identified labels and any edge connected to the node identified by any of the one or more identified labels. As a result, the information processing device 100 can accurately identify one or more edges that form a diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information.

[0196] The information processing device 100 can output structural information that shows the structure of a diagram, which is formed by one or more identified labels, the nodes identified by each of the one or more identified labels, and one or more identified edges. This allows the information processing device 100 to make the structure of the diagram accessible externally.

[0197] According to the information processing device 100, based on the matching results, it is possible to identify any of the acquired second labels that cannot be associated with any of the acquired first labels. According to the information processing device 100, the identified second label can be associated with any of the identified edges on the diagram image whose distance to that second label is below a threshold, and this can be output. This makes it easier for external parties to understand the meaning of the edges.

[0198] The information processing device 100 can generate image information showing an image of a diagram drawn at a predetermined resolution. This allows the information processing device 100 to improve the accuracy of identifying one or more first labels that identify each of the different nodes included in the diagram when analyzing the image information.

[0199] According to the information processing device 100, based on image information, an image can be divided such that the number of diagram components drawn overlapping at the boundaries dividing the image can be reduced. According to the information processing device 100, one or more first labels can be obtained by analyzing each divided part of the image based on the image information. As a result, the information processing device 100 can improve the accuracy of identifying one or more first labels that identify different nodes included in the diagram when analyzing image information. The information processing device 100 can make it easier to analyze the image indicated by the image information, even if the image size is large.

[0200] According to the information processing device 100, one or more first labels can be obtained by analyzing image information. As a result, the information processing device 100 can obtain one or more first labels without the need for another computer, and the structure of the diagram can be identified.

[0201] According to the information processing device 100, by analyzing document information, one or more second labels and one or more first edges can be obtained. As a result, the information processing device 100 can obtain one or more second labels and one or more first edges without the need for another computer, and the structure of the diagram can be identified.

[0202] According to the information processing device 100, based on the matching results, one or more labels can be identified by removing any of the acquired second labels that cannot be associated with any of the acquired first labels. As a result, the information processing device 100 can accurately identify one or more labels that identify different nodes forming a diagram by appropriately combining the results of analyzing the image information and the results of analyzing the document information.

[0203] The information processing method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a PC or workstation. The information processing program described in this embodiment is recorded on a computer-readable recording medium and executed by being read from the recording medium by the computer. The recording medium can be a hard disk, flexible disk, CD (Compact Disc)-ROM, MO (Magneto Optical Disc), DVD (Digital Versatile Disc), etc. Furthermore, the information processing program described in this embodiment may be distributed via a network such as the Internet.

[0204] 100 Information Processing Device 101 Diagram 110, 1210 Image Information 111 First Label 120, 900, 1100, 1220 Document Information 121 Second Label 122 First Edge 131 Label 132, 602, 702 Edge 200 Information Processing System 201 Image Analysis Device 202 Document Analysis Device 203 Client Device 210 Network 300 Bus 301 CPU 302 Memory 303 Network I / F 304 Recording Medium I / F 305 Recording Medium 306 Display 307 Input Device 400 Storage Unit 401 Acquisition Unit 402 Image Analysis Unit 403 Document Analysis Unit 404 Correspondence Relationship Identification Unit 405 Label Identification Unit 406 Edge Identification Unit 407 Structure Identification Unit 408 Output Unit 500 Large-scale language model 501 Text input unit 502 Image conversion unit 503 Image segmentation unit 504, 507 Node recognition unit 505, 508 Edge recognition unit 506 Document extraction unit 509 Correspondence processing unit 510 Filtering unit 511 Label correction unit 512 Edge correction unit 513 Semantic correction unit 514 Task processing 600 Screen system diagram 601, 701 Nodes 610, 710, 711, 720 Components 700, 1600 Sequence diagram 800, 1000 Original data 1300 Screen 1310, 1401, 1402 Images 1410, 1420, 1510, 1520, 1610 Recognition result table

Claims

1. An information processing program characterized by causing a computer to perform the following processes:

1. Obtain one or more first labels that identify nodes included in a diagram, obtained by analyzing image information of a diagram; 2. Obtain one or more second labels that identify nodes included in a diagram, and one or more first edges included in a diagram, obtained by analyzing document information of a diagram; 3. Identify one or more labels that identify nodes forming the diagram based on the result of associating any of the obtained first labels with any of the obtained second labels; and 4. Identify one or more edges that form the diagram from among the obtained first edges based on the identified one or more labels.

2. The information processing program according to claim 1, characterized in that the process of identifying one or more labels involves identifying one or more labels by correcting the acquired one or more first labels based on the result of associating a first label of any of the acquired one or more first labels with a second label of any of the acquired one or more second labels, wherein the similarity is greater than or equal to a threshold.

3. The information processing program according to claim 2, characterized in that the process of identifying one or more labels is to correct any of the acquired first labels based on any of the acquired second labels that are associated with that first label.

4. The information processing program according to claim 3, characterized in that the process of identifying one or more edges involves identifying, from among the acquired one or more first edges, a node identified by any of the identified one or more labels, and an edge connected to the node identified by any of the identified one or more labels.

5. The information processing program according to claim 4, characterized in that it causes the computer to perform a process that outputs structural information indicating the structure of the diagram, which is formed by the one or more identified labels, the nodes identified by each of the one or more identified labels, and the one or more identified edges.

6. The information processing program according to claim 5, characterized in that, based on the results of the association, it further outputs an information processing program that associates any of the acquired second labels that cannot be associated with any of the acquired first labels with any of the identified edges on the image in which the diagram is drawn, where the distance to the said second label is less than or equal to a threshold.

7. An information processing program according to any one of claims 1 to 6, characterized in that it causes the computer to perform a process to generate image information showing the diagram drawn at a predetermined resolution.

8. An information processing program according to claim 7, characterized in that the computer is made to perform a process of dividing the image based on the image information such that the number of diagram components drawn overlapping at the boundaries dividing the image is reduced, and the process of obtaining the one or more first labels is performed by analyzing the divided part of the image based on the image information to obtain the one or more first labels.

9. The information processing program according to claim 1, characterized in that the process of obtaining one or more first labels is to obtain one or more first labels by analyzing the image information.

10. The information processing program according to claim 1, characterized in that the process of obtaining the one or more second labels and the one or more first edges is performed by analyzing the document information to obtain the one or more second labels and the one or more first edges.

11. The information processing program according to claim 1, characterized in that the process of identifying one or more labels involves identifying one or more labels by removing any of the acquired second labels that cannot be associated with any of the acquired first labels, based on the result of associating any of the acquired first labels with any of the acquired second labels.

12. An information processing method characterized in that a computer performs the following steps: obtain one or more first labels that identify nodes included in the diagram by analyzing the image information of the diagram; obtain one or more second labels that identify nodes included in the diagram and one or more first edges included in the diagram by analyzing the document information of the diagram; identify one or more labels that identify nodes forming the diagram based on the result of associating any of the obtained first labels with any of the obtained second labels; and identify one or more edges that form the diagram from the obtained first edges based on the identified one or more labels.

13. An information processing apparatus characterized by having a control unit that: obtains one or more first labels that identify nodes included in a diagram by analyzing image information of a diagram; obtains one or more second labels that identify nodes included in a diagram and one or more first edges included in a diagram by analyzing document information of a diagram; identifies one or more labels that identify nodes forming the diagram based on the result of associating any of the obtained first labels with any of the obtained second labels; and identifies one or more edges that form the diagram from among the obtained first edges based on the identified one or more labels.

Images