Information processing device, information processing method, program, and recording medium

Abstract

This information processing device comprises: a storage means that stores, in a storage unit as document information in association with an item to be extracted, an item name and item content corresponding to the item to be extracted, the item name and item content having been extracted from a document processed in the past on the basis of the item to be extracted, and position information indicating each position of the item name and the item content; a retrieval means that retrieves, from among the document information stored in the storage unit on the basis of a character string included in a document to be processed and position information indicating a position of the character string, the document information of a document processed in the past and corresponding to the document to be processed; and an information extraction means that inputs, to a large language model, a prompt including the retrieved document information to perform inference by the large language model, and extracts, from the document to be processed, an item name and item content corresponding to the item to be extracted.

Description

Information Processing Apparatus, Information Processing Method, Program, and Recording Medium

[0001] The present invention relates to an information processing apparatus, an information processing method, a program, and a recording medium.

[0002] Techniques for extracting handwritten or printed character strings from image data such as documents using optical character recognition (OCR) processing are known. There is also a technique for performing optical character recognition processing on image data such as documents and extracting character strings corresponding to specified items. As a method for extracting a character string corresponding to a specified item, for example, there is a method of extracting a character string located in a predetermined direction with respect to an item name in a document, or a method of extracting a character string near an item name included in image data as a candidate character string and determining and extracting one character string from the candidate character strings based on the number of occurrences in the image data.

[0003] Japanese Patent Application Laid-Open No. 2023-46684

[0004] However, when the item name related to the item to be extracted is not included in the document, or when the item name specified as the item to be extracted is different from the item name included in the document, the content of the item corresponding to the item (hereinafter also referred to as "item content") may not be accurately extracted. Also, in a document, when the item name or item content corresponding to the item to be extracted is described across multiple lines or when the item content is described in a segmented manner, the item name and item content may not be accurately extracted. An object of the present invention is to provide an information processing apparatus, an information processing method, a program, and a recording medium capable of accurately extracting an item name and item content corresponding to an item to be extracted from image data of a document.

[0005] The information processing device according to the present invention is characterized by comprising: a storage means for storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted; a search means for searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction means for inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

[0006] According to the present invention, it is possible to provide an information processing device, an information processing method, a program, and a recording medium that can accurately extract item names and item contents corresponding to target items from image data of a document.

[0007] Figure 1 is a diagram showing an example of the hardware configuration of an information processing device. Figure 2 is a diagram showing an example of the functional configuration of an information processing device. Figure 3 is a flowchart showing an example of processing by an information processing device. Figure 4 is a diagram illustrating the information stored in the document information DB. Figure 5A is a diagram showing an example of a document to be processed. Figure 5B is a diagram illustrating an example of the information stored in the document information DB. Figure 6A is a diagram illustrating information extraction from a document. Figure 6B is a diagram illustrating information extraction from a document. Figure 7 is a diagram illustrating an example of prompts related to information extraction. Figure 8A is a diagram showing an example of a UI that displays the extraction results before modification. Figure 8B is a diagram showing an example of a UI that displays the modified extraction results. Figure 9A is a diagram illustrating modification of the extraction results. Figure 9B is a diagram illustrating modification of the extraction results. Figure 10A is a diagram illustrating an example of prompts related to information extraction. Figure 10B is a diagram illustrating an example of prompts related to information extraction. Figure 10C is a diagram illustrating an example of prompts related to information extraction. Figure 11 is a diagram illustrating an example of information extraction results.

[0008] Embodiments of the present invention will be described based on the drawings.

[0009] Figure 1 shows an example of the hardware configuration of an information processing device 100 in one embodiment of the present invention. The information processing device 100 includes a CPU 101, ROM 102, RAM 103, auxiliary storage device 104, output device 105, input device 106, and network I / F 107. The CPU 101, ROM 102, RAM 103, auxiliary storage device 104, output device 105, input device 106, and network I / F 107 are communicated together via a system bus 108.

[0010] The CPU (Central Processing Unit) 101 is a central processing unit that controls various operations of the information processing device 100. For example, the CPU 101 may control the operation of the entire information processing device 100. The ROM (Read Only Memory) 102 stores control programs, boot programs, etc., that can be executed by the CPU 101. The RAM (Random Access Memory) 103 is the main memory of the CPU 101 and is used as a work area or a temporary storage area for deploying various programs.

[0011] The auxiliary storage device 104 stores various data and programs. For example, the auxiliary storage device 104 is implemented by a storage device capable of temporarily or permanently storing various data, such as an HDD (Hard Disk Drive) or non-volatile memory represented by an SSD (Solid State Drive).

[0012] The output device 105 is a device that outputs various types of information and is used to present various types of information to the user. For example, the output device 105 may be implemented by a display device such as a display. The output device 105 may present information to the user by displaying various types of display information. As another example, the output device 105 may be implemented by an acoustic output device that outputs sounds such as voice or electronic sounds. In this case, the output device 105 may present information to the user by outputting sounds such as voice or electronic sounds. Furthermore, the device to which the output device 105 is applied may be appropriately changed depending on the medium used to present information to the user.

[0013] The input device 106 is used to receive various instructions from the user. For example, the input device 106 may include input devices such as a mouse, keyboard, or touch panel. As another example, the input device 106 may include a sound collection device such as a microphone to collect the voice spoken by the user. In this case, various analysis processes such as acoustic analysis and natural language processing may be applied to the collected voice so that the content of the voice is recognized as an instruction from the user. Furthermore, the device applied as the input device 106 may be changed as appropriate depending on the method of recognizing the user's instructions. In addition, multiple types of devices may be applied as the input device 106.

[0014] The network interface 107 is used for communication with external devices via a network. The device used as the network interface 107 may be changed as appropriate depending on the type of communication path and the applicable communication method.

[0015] The CPU 101 loads the program stored in the ROM 102 or auxiliary storage device 104 into the RAM 103 and executes the program, thereby realizing the various functions and processes of the information processing device described later. The program for the information processing device 100 may be provided to the information processing device 100 by a recording medium such as a CD-ROM, or it may be downloaded via a network or the like. When the program for the information processing device 100 is provided by a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 104 when the recording medium is set in a predetermined drive device.

[0016] The configuration shown in Figure 1 is merely an example and does not necessarily limit the hardware configuration of the information processing device 100 in this embodiment. For example, some components such as the output device 105 and the input device 106 may be omitted. Another example is that configurations may be added as appropriate depending on the functions to be realized by the information processing device 100.

[0017] Figure 2 shows an example of the functional configuration of the information processing device 100 in this embodiment. The information processing device 100 includes a control unit 201, an input / output control unit 202, a storage unit 203, an acquisition unit 204, a search unit 205, an instruction generation unit 206, a first information extraction unit 207, a modification unit 208, and a second information extraction unit 209.

[0018] The control unit 201 is responsible for controlling each component of the information processing device 100. The input / output control unit 202 performs various processes related to presenting various information to the user and receiving information input from the user (e.g., instructions). For example, the input / output control unit 202 may perform processes related to presenting a UI (User Interface) and processes related to receiving input via the UI. This enables the information processing device 100 to recognize instructions from the user and present the user with the results of processing corresponding to those instructions.

[0019] The memory unit 203 schematically represents a memory area for storing various data and programs. For example, the memory unit 203 may store data and programs for each component of the information processing device 100 to execute processing. The memory unit 203 may also store a trained model that has undergone machine learning (deep learning) and is used for inference related to information extraction performed in the information processing device 100. Furthermore, the memory unit 203 may store various information related to documents, such as document information and information extraction results.

[0020] In this embodiment, document information refers to information that associates the item name, the content of the item corresponding to the item name, and the positional information for the item name and item content, respectively, with the item to be extracted, based on the item to be extracted (item to be extracted). The positional information refers to information that indicates the position of the string representing the item name and item content within the document.

[0021] The acquisition unit 204 acquires image data of a document by optically scanning or photographing it. The acquisition unit 204 also performs optical character recognition (OCR) processing on the acquired image data of the document to acquire the string of characters contained in the image data of the document and the position information related to that string of characters. The acquisition unit 204 may also accept and acquire image data of a document that has been scanned or photographed externally in advance as input, without scanning or photographing the document itself. Alternatively, the acquisition unit 204 may accept and acquire the string of characters contained in the document and their position information obtained as a result of external optical character recognition processing as input.

[0022] The search unit 205 searches the document information database (document information DB), which stores and stores document information about documents processed in the past, based on the information obtained from the document to be processed by the acquisition unit 204 (hereinafter also referred to as the "target document"), and searches for documents that are similar in format to the target document from among the documents that have been processed in the past. Specifically, the search unit 205 searches the document information DB based on information about item names and searches for documents that have a high degree of matching with the target document in terms of the strings contained in the document and their positional information, as documents that correspond to the target document and are similar in format to the target document. The previously processed documents that are similar in format to the target document, obtained as a search result, are referenced in the information extraction process for the target document.

[0023] In the information processing device 100 of this embodiment, the instruction generation unit 206 and the first information extraction unit 207 can use a Large Language Model (LLM) to extract information about the specified items to be extracted from the image data of the document. For example, the instruction generation unit 206 generates instructions for extracting various information about the items to be extracted from the target document and inputs them to the first information extraction unit 207, and the first information extraction unit 207 uses the Large Language Model (LLM) to extract information according to the input instructions. The Large Language Model (LLM) is a language model constructed using a large amount of text data (large corpus, etc.) and deep learning technology. When text data called prompts indicating instructions are input, it performs inference based on the prompts and generates and outputs text data corresponding to the input prompts.

[0024] The instruction generation unit 206 generates a prompt that includes instructions for extracting various information about the items to be extracted from the target document, a string of characters and their positional information contained in the image data of the target document, and inputs it to the first information extraction unit 207. In this embodiment, the various information about the items to be extracted from the target document includes the item name (string of characters) corresponding to the item to be extracted, the item content (string of characters) corresponding to the item name, and the positional information (coordinate information) of the item name and item content.

[0025] The first information extraction unit 207 performs inference using a large-scale language model (LLM) based on prompts generated and input by the instruction generation unit 206, and extracts various information related to the items to be extracted as instructed by the prompts from the image data of the target document. The modification unit 208 receives a modification request from the user regarding the extraction results of information about the target document by the first information extraction unit 207, and modifies the extraction results according to the modification request.

[0026] Furthermore, in the information processing device 100 of this embodiment, the second information extraction unit 209 can use document information from documents processed in the past to extract information about the specified target items from the image data of the documents, based on the location information of the item names and item contents. The second information extraction unit 209 extracts various information about the specified target items from the image data of the target document based on the location information of the item names and item contents in the document information of documents processed in the past. Specifically, the second information extraction unit 209 searches for documents processed in the past and refers to documents with a similar format to the target document, and extracts various information about the target items from the target document based on the location information of the item names and item contents in those documents.

[0027] Furthermore, in inference using a Large-Scale Language Model (LLM), there is a method called few-shot learning that improves the accuracy of responses by taking advantage of the characteristic of the LLM that it has a high accuracy in returning output adapted to the presented example (sample) by including an example (sample) in the prompt. Therefore, the instruction generation unit 206 may generate a prompt using document information of documents highly similar to the target document (for example, documents obtained by searching in the search unit 205, or documents whose similarity to the target document in terms of document format is above a threshold) as an example for few-shot learning, and input it to the first information extraction unit 207. In this way, by providing the LLM with information of documents highly similar to the target document as an example for few-shot learning, it is possible to improve the accuracy of extracting item names and item contents corresponding to the items to be extracted from the target document.

[0028] Referring to Figure 3, the processing in the information processing device 100 in this embodiment will be described. Figure 3 is a flowchart showing an example of processing in the information processing device 100 in this embodiment.

[0029] In step S301, the acquisition unit 204 scans or photographs the document to be processed (target document) to acquire image data, performs optical character recognition processing on the acquired image data, and acquires the string of characters and their positional information contained in the image data of the target document. Here, the positional information of the string of characters is acquired as four values: the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate of the string of characters in an XY coordinate system with one endpoint of the document (e.g., the top-left or bottom-left point) as the origin. In addition, during the optical character recognition processing, information indicating the width (horizontal length) and height (vertical length), which are the dimensions of the document (corresponding to the maximum x-coordinate and maximum y-coordinate values ​​that the document can take), is acquired.

[0030] In step S302, the search unit 205 searches the document information database to obtain documents from previously processed documents that are similar in format to the target document, based on information regarding item names in the document information. The document information database is provided, for example, in the storage unit 203 and stores document information of documents that have been processed in the past.

[0031] Figure 4 illustrates the document information of previously processed documents stored in the document information database. Document information for previously processed documents is stored in the document information database, for example, for each page of the document. The document information may include the document type, a uniquely assigned identifier (ID), document information, extraction results, and information indicating the document's width and height. The extraction results may include, for example, the item name indicating the item to be extracted, whether or not information about that item exists, the content of the item name extracted from the document, the string where the item name existed in the document (a pair of the string and its position information), the position information of the item name, and information about the item. The string where the item name existed, the position information of the item name, and the information about the item are in a list structure, allowing for the storage of multiple pieces of information. The information about the item also includes the content of the item extracted from the document, the string where the item content existed in the document (a pair of the string and its position information), and the position information of the item. The string where the item content existed and the position information of the item are also in a list structure, allowing for the storage of multiple pieces of information. For example, if an item name or information corresponding to a single item in a document spans multiple lines or is separated by lines, multiple pieces of information are stored as strings representing the item name and item content. These combined strings are then stored as the item name and item content. Furthermore, the position information of the item name and item is normalized using the document's width and height so that the values ​​are within the range of 0 to 1. By storing this normalized position information, it is possible to restore the position information of strings that match the size of the document, even for documents of different sizes. In this embodiment, the item name, the content of the item corresponding to the item name, and the position information for both the item name and item content, extracted from the document based on the item to be extracted, are all stored as document information associated with the item to be extracted. In the example shown in Figure 4, there is one item to be extracted (one item name indicating the item to be extracted), but this is just an example, and the number of items to be extracted can be arbitrary.

[0032] As an example, Figure 5B shows an example of document information extracted from the document shown in Figure 5A and stored in the document information database. Figure 5B shows document information extracted from the document information of the document shown in Figure 5A, with "Deliverables" as the item to be extracted and stored. In the document shown in Figure 5A, the information written in the "Deliverables" column 501 is extracted as the information corresponding to the item to be extracted, "Deliverables," and stored in the document information database. As shown in Figure 5B, since the information corresponding to the item to be extracted, "Deliverables," exists in the document, the presence or absence of the information is stored as "Yes." In addition, "Deliverables" is stored as the content of the item name corresponding to the item to be extracted, and the string indicating the existence of the item name is "Deliverables" and its position information, "Deliverables: A, A, A, A." "A', A', A', A'" is stored as the position information of the item name. Here, “A', A', A', A'” is the value obtained by normalizing “A, A, A, A” using the width and height of the document, i.e., “A', A', A', A'” = “A / (width), A / (height), A / (width), A / (height)”. In addition, the contents of this item include “DB design document”, “similar function study result report”, “new and old business function comparison table”, and “main table relationship diagram”. The strings containing the content of each item, including the strings of each item and their positional information, are stored as follows: "DB Design Document: B, B, B, B", "Similar Function Study Results Report: C, C, C, C", "Business Function New vs. Old Comparison Table: D, D, D, D", and "Main Table Relationship Diagram: E, E, E, E". The positional information of each item name is normalized based on the size of the document.

[0033] In step S302, when searching the document information DB, the search unit 205 compares the strings and their positional information of each item name in previously processed documents with the strings and their positional information of the target document obtained in step S301, thereby searching for documents with a similar format to the target document from among previously processed documents. Since item names are expected to be in the same position in documents of the same format, by comparing the item names, it is possible to search for documents with a similar format to the target document from among previously processed documents. For example, the search unit 205 determines whether the strings exist in the same position in the target document for all item names included in the document information of a given previously processed document. For example, the search unit 205 determines that the strings exist in the same position if the center point of a rectangle calculated based on the positional information of the strings included in the target document exists within a rectangle based on the positional information of the item names in previously processed documents (more specifically, a rectangle based on the positional information after conversion to match the size of the target document). In this way, if the center point of a rectangle calculated based on the position information of the strings contained in the target document is within a rectangle based on the position information of the item name, it is determined that the string exists at the same position. This makes it possible to treat even if the position of the string is slightly shifted as the same position. If it is determined that the string exists at the same position in the target document for all item names, the search unit 205 determines whether those strings match the item name strings for each position. As a result, if all of the strings match the item name strings, that is, if all the strings related to the item names and the positions in which they exist match, it is determined that the previously processed document is a document with a similar format to the target document. This search process is performed on all previously processed documents stored in the document information DB. Alternatively, previously processed documents stored in the document information DB may be classified according to attributes such as the type of document, and the above search process may be performed only on documents classified according to the attributes corresponding to the target document. Alternatively, the process in step S302 may be terminated when a document with a similar format to the target document is found among the previously processed documents, and the process in step S303 may be executed.

[0034] In step S303, the control unit 201 determines, based on the results of searching the document information DB in step S302, whether there are any documents with the same format as the target document among the documents processed in the past. If the control unit 201 determines that there are documents with the same format as the target document (YES), the process in step S304 is executed. On the other hand, if the control unit 201 determines that there are no documents with the same format as the target document (NO), the process in step S308 is executed.

[0035] In step S304, the control unit 201 determines whether there are any items among the items to be extracted from the target document that may contain multiple pieces of information as item content, based on previously processed documents that are similar in format to the target document. To determine whether an item may contain multiple pieces of information, for example, one can pre-set whether an item may contain multiple pieces of information as item content for each item, and then determine whether there are any items that may contain multiple pieces of information based on that information. If the control unit 201 determines that there are items that may contain multiple pieces of information (YES), the process in step S305 is executed. On the other hand, if the control unit 201 determines that there are no items that may contain multiple pieces of information (NO), the process in step S305 is not executed, and the process in step S306 is executed.

[0036] The reason for determining whether or not an item among the items to be extracted may contain multiple pieces of information as its content will be explained with reference to Figures 6A and 6B. Let Figure 6A be a previously processed document that has been determined to be similar in format to the target document, and Figure 6B be the target document. For example, when extracting information by matching location information as will be described later, let's assume that information is extracted if the center point of the bounding rectangle for the string calculated based on the location information of the string in the target document is included in the bounding rectangle of the string in the previously processed document. In this case, the center point 615 of the bounding rectangle 611 for the string "DB design document" and the center point 620 of the bounding rectangle 619 for the string "hand-delivered" in the target document shown in Figure 6B are included in the bounding rectangles 601 and 602 for the string in the previously processed document shown in Figure 6A, so information regarding the strings "DB design document" and "hand-delivered" can be extracted based on location information. On the other hand, the center point 616 of the bounding rectangle 612 for the string "Report on Similar Functions," the center point 617 of the bounding rectangle 613 for the string "Comparison Table of Old and New Business Functions," and the center point 618 of the bounding rectangle 614 for the string "Main Table Relationship Diagram" in the target document shown in Figure 6B are not included in any of the bounding rectangles for strings in the previously processed document shown in Figure 6A. Therefore, this information is not extracted based on location information. In order to prevent information omissions and acquisition errors in this way, it is determined whether or not an item can contain multiple pieces of information as item content, and whether or not to extract the information based on location information.

[0037] Returning to Figure 3, in step S305, the instruction generation unit 206 and the first information extraction unit 207 use a large-scale language model (LLM) to extract item names and item contents corresponding to items to be extracted, which may contain multiple pieces of information, from the target document. Specifically, the instruction generation unit 206 generates a prompt to be given to the large-scale language model (LLM) for extracting information of items that may contain multiple pieces of information from the target document. The instruction generation unit 206 generates a prompt that includes an instruction to extract information corresponding to items to be extracted, which may contain multiple pieces of information, from the target document, as well as the string of the target document and its position information. As an example for few-shot learning, the instruction generation unit 206 further generates a prompt that includes information of previously processed documents (the string of the document and its position information, and the extraction results) which have been determined to be similar in format to the target document.

[0038] An example of a prompt generated by the instruction generation unit 206 is shown in Figure 7. As shown in Figure 7, the prompt 700 includes a description of how to extract information from the document 710, a group of item names and descriptions to be extracted 720, a string of previously processed documents and their location information and an example of a response 730, an example of an output format 740, and a string of the target document and its location information 750. The description of the extraction method 710 is a description of how to extract information on the items to be extracted from the target document, and the group of item names and descriptions to be extracted 720 is the item name and description for one or more items to be extracted. By including this item description in the prompt, it is possible to update the knowledge of items held by the Large-Scale Language Model (LLM) and to train the Large-Scale Language Model (LLM) to learn knowledge of items it does not hold, enabling accurate extraction of information corresponding to the specified items. The strings of previously processed documents, their location information, and example responses 730 are examples (samples) for few-shot learning, and include, for example, information (strings of documents and their location information, and extraction results) 731 of previously processed documents that were determined to be similar in format to the target document. The example output format 740 is an example of the output format when outputting information corresponding to the items to be extracted as extraction results. The strings of the target document and their location information 750 are the strings and their location information contained in the target document acquired in step S301, and include the strings and their location information 751 of the pages of the target document.

[0039] After the instruction generation unit 206 generates a prompt to be given to the large-scale language model (LLM), the first information extraction unit 207 inputs the generated prompt to the large-scale language model (LLM) to perform inference regarding information extraction and extracts information from the target document that corresponds to the item to be extracted, which may contain multiple pieces of information.

[0040] In step S306, the second information extraction unit 209 extracts information from the target document that corresponds to the remaining items of the item to be extracted (items that cannot contain multiple pieces of information as item content) based on the location information of previously processed documents that have been determined to be similar in format to the target document, and the location information of the target document. For example, the second information extraction unit 209 extracts a string of characters in which the center point of a rectangle calculated based on the location information obtained in step S301 lies within a rectangle based on the location information of an item in a previously processed document, as the information (item content) corresponding to the item to be extracted. If the location information of an item contains multiple pieces of information, the information can be extracted using a rectangle that contains all of the corresponding item content strings, that is, a rectangle defined by the minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate within that location information.

[0041] In step S307, the control unit 201 outputs the extraction results of target documents corresponding to the items to be extracted obtained in steps S305 and S306. The control unit 201 may also output the strings and their position information in the target documents acquired in step S301. After executing the process in step S307, the information processing device 100 terminates the process shown in Figure 3.

[0042] In the explanation above, a Large-Scale Language Model (LLM) is used to extract item names and item contents corresponding to items to be extracted that may contain multiple pieces of information. However, it is also possible to use the Large-Scale Language Model (LLM) to extract item names and item contents corresponding to all items to be extracted, not just items that may contain multiple pieces of information. Furthermore, regardless of whether or not there are items that may contain multiple pieces of information, as an example for few-shot learning, the Large-Scale Language Model (LLM) may be given a prompt containing information from previously processed documents that have been determined to be similar in format to the target document, and the Large-Scale Language Model (LLM) may be used to extract item names and item contents corresponding to items to be extracted.

[0043] In step S308, which is executed when the control unit 201 determines in step S303 that there is no document in the past processed documents that has the same format as the target document, the instruction generation unit 206 and the first information extraction unit 207 extract information corresponding to each item to be extracted from the target document using a large language model (LLM). Specifically, as shown by an example in FIG. 7, the instruction generation unit 206 generates a prompt for extracting information corresponding to each item to be extracted from the target document and providing it to the large language model (LLM). At this time, the instruction generation unit 206 may include, for example, an example of a document showing how to extract information from a document and an example of a past processed document having a format similar to the target document as an exemplification for few-shot learning to generate the prompt. The first information extraction unit 207 inputs the prompt generated by the instruction generation unit 206 to the large language model (LLM) to perform inference regarding information extraction, and extracts information corresponding to the item to be extracted from the target document. Here, in the present embodiment, as information corresponding to the item to be extracted, a character string regarding the item name and item content corresponding to the item to be extracted and its position information are extracted. That is, the content of the item name corresponding to the item to be extracted, the character string (and its position information) where the item name exists, and the position information of the item name, and the content of the item, the character string (and its position information) where the item exists, and the position information of the item are extracted from the target document as information corresponding to the item to be extracted. In addition to these information, other information may be extracted from the target document.

[0044] In step S309, the control unit 201 outputs the extraction result of the target document corresponding to the item to be extracted obtained by the process of step S308. Note that the control unit 201 may output in combination the character string and its position information in the target document acquired in step S301.

[0045] In step S310, the control unit 201 determines whether or not there is any modification input for the extraction results of the target document output in step S309. The control unit 201 determines that there is modification input if, for example, a modification request is confirmed from a user who has confirmed the extraction results of the output target document. If the control unit 201 determines that there is modification input for the extraction results (YES), the process in step S311 is executed. On the other hand, if the control unit 201 determines that there is no modification input for the extraction results (NO), the process in step S311 is not executed, and the process in step S312 is executed.

[0046] In step S311, the modification unit 208 modifies the extraction results of the target document according to the input for modification of the extraction results. In step S312, the control unit 201 associates the string and its position information related to the target document with the extraction results and stores them in the document information DB. For example, the control unit 201 assigns a unique identifier (ID) to the target document, and stores the identifier (ID), the string and its position information obtained in step S301, and the extraction results obtained in step S308 (or the modified extraction results modified in step S311) in the document information DB. After executing the process in step S312, the information processing device 100 terminates the process shown in Figure 3.

[0047] Referring to Figures 8A, 8B, 9A, and 9B, the correction process for the extraction results of the target document obtained by extracting information using a Large-Scale Language Model (LLM) will be explained. Figure 8A shows an example of UI 800 that displays the extraction results before correction. In the example shown in Figure 8A, "DB design document" is extracted as the item content corresponding to the item "deliverables" to be extracted, and "store the electronic file on our designated server" is extracted as the item content corresponding to the item "delivery method" to be extracted. In the image of the target document, the extracted item content is displayed in an identifiable manner using borders 801 and 811. Also, the "deliverables" button 820 and the "delivery method" button 830 corresponding to the items are displayed. The extraction results before correction corresponding to this are shown in Figure 9A. In this case, only "DB design document" is extracted as the item content corresponding to the item "deliverables" to be extracted, and "similar function study result report," "business function old and new comparison table," and "main table relationship diagram" are not extracted as item content.

[0048] The user can modify the extraction result by performing an input operation related to the modification on the UI where the extraction result is displayed. For example, on the UI where the extraction result is displayed, the extraction result can be modified by adding or moving a frame line. As an example, when adding "Similar Function Consideration Result Report", "Business Function New and Old Comparison Table", and "Main Table Related Diagram" as the item contents corresponding to the item "Deliveries" to be extracted, as shown in FIG. 8B, on the UI 850 where the extraction result is displayed, in the image of the target document, after adding frame lines 851, 852, and 853 so that the character strings of "Similar Function Consideration Result Report", "Business Function New and Old Comparison Table", and "Main Table Related Diagram" are respectively included, the "Deliveries" button 820 is pressed. Thereby, the modification unit 208 modifies, for example, as shown in FIG. 9B, in addition to the "DB Design Document", the item contents corresponding to the item "Deliveries" to be extracted to include "Similar Function Consideration Result Report", "Business Function New and Old Comparison Table", and "Main Table Related Diagram". In the above-described example, the modification of the item contents has been described, but the modification of the item name may also be performed in the same manner. In this way, the user can modify the character string and position information regarding the item name and item contents corresponding to the item to be extracted in the extraction result only by performing an operation on the UI where the extraction result is displayed. In the example of the UI shown in FIGS. 8A and 8B, the information included in the extraction result is displayed so as to be distinguishable by a frame line, but it is not limited thereto. For example, highlighting, selection display, etc. may be used, and display methods used in general UIs are applicable.

[0049] In this embodiment, specific examples of prompts generated by the instruction generation unit 206 are shown in Figures 10A to 10C. The elements shown in Figures 10A to 10C are combined to form a single prompt. In Figures 10A to 10C, element 1010 corresponds to the extraction method explanation 710, and element 1020 corresponds to the item name to be extracted and the item explanation group 720. Element 1030 corresponds to the document string, location information, and example answer 730. Element 1040 corresponds to the output format example 740, and element 1050 corresponds to the target document string and location information 750. By providing the prompts shown in Figures 10A to 10C to the Large-Scale Language Model (LLM), inference regarding information extraction is performed, and the extraction results shown in Figure 11 are obtained.

[0050] According to this embodiment, the information processing device 100 searches for document information of previously processed documents that correspond to the target document, based on the string of characters and their positional information contained in the target document. Then, using the document information of the documents corresponding to the target document obtained through the search, it extracts the item name and item content corresponding to the item to be extracted, thereby enabling accurate extraction of the item name and item content corresponding to the item to be extracted from the target document.

[0051] As explained above, the determination of whether a string exists at the same location in the target document and a previously processed document is made by checking whether the center point of a rectangle calculated based on the location information of the string contained in the target document exists within a rectangle based on the location information of the previously processed document. However, this is not the only way to determine whether a string exists at the same location in the target document and a previously processed document. For example, if the distance between the center point of a rectangle calculated based on the location information of a previously processed document and the center point of a rectangle calculated based on the location information of the string contained in the target document is less than or equal to a predetermined threshold, it may be determined that the strings exist at the same location. Alternatively, for example, if the IoU (Intersection over Union) between a rectangle based on the location information of a previously processed document and a rectangle based on the location information of the string contained in the target document is greater than or equal to a predetermined threshold, it may be determined that the strings exist at the same location. Furthermore, the string whose distance between the center point of a rectangle calculated based on the location information of a previously processed document and the center point of a rectangle calculated based on the location information of the string contained in the target document is shortest may be considered the string that exists at the same location. Alternatively, from a predetermined number of strings, starting from the shortest distance, the string with the closest cosine similarity may be considered as existing at the same location. Alternatively, images of the vicinity of a string may be input into a large-scale language model to determine whether or not a string exists at the same location.

[0052] Furthermore, the explanation above describes how to extract various information about the target item by inputting strings extracted from images and information from previously processed documents. However, the method is not limited to this; information about the target item may also be extracted by inputting the acquired image (hereinafter referred to as the "current image") and images of previously processed documents (hereinafter referred to as "past images") along with strings extracted from images and information from previously processed documents using other methods. In addition, to distinguish between the current image and past images, each image may be assigned a string that identifies it before input. This makes it possible to accurately extract various information about the target item using a Large-Scale Language Model (LLM).

[0053] Furthermore, as described above, a Large-Scale Language Model (LLM) is used to extract various information about the items to be extracted. This makes it possible to accurately extract item names and item contents even if the target document contains multiple strings corresponding to item names and item contents. In addition, as described above, the prompts input to the Large-Scale Language Model (LLM) include information including item descriptions and extraction results from previously processed documents. This makes it possible to accurately extract items from the target document as items to be extracted, even if the item names in the target document differ from those in the target document, by learning from information in previously processed documents.

[0054] Furthermore, as explained above, if the string for the item to be extracted cannot contain multiple pieces of information as item content (i.e., it contains only a single piece of information as item content), the item to be extracted is not used with a Large-Scale Language Model (LLM), but rather with location information in the target document and location information in previously processed documents. This improves processing speed and reduces processing load (computational load) compared to using a Large-Scale Language Model (LLM).

[0055] <Other Embodiments> The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various recording media; and having the computer (or CPU or MPU, etc.) of the system or device read and execute the program. Computer-readable recording media on which the program is recorded and computer program products such as the program can also be applied as embodiments of the present invention. Examples of recording media that can be used include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0056] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0057] The following devices and methods are also included in the disclosure of this embodiment: (1) An information processing apparatus characterized by comprising: a storage means for storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, extracted from documents processed in the past based on the item to be extracted; a search means for searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction means for inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed. (2) The information processing apparatus according to (1), wherein the search means compares the item name and position information of the item name contained in the document information stored in the storage unit with a string contained in the document to be processed and position information of the string, and searches for the document information of a document corresponding to the document to be processed. (3) The information processing device according to (1) or (2), wherein the information extraction means, if the item to be extracted is an item that may contain multiple items, uses the large-scale language model to extract the item name and item content corresponding to the item to be extracted from the document to be processed, and if the item to be extracted is an item that contains only one item content, extracts the item name and item content corresponding to the item to be extracted from the document to be processed based on the location information contained in the document information obtained by the search and the location information in the document to be processed. (4) The information processing device according to (3), wherein the information extraction means, if the item to be extracted is an item that contains only one item content, performs a comparison using the location information contained in the document information obtained by the search and the location information in the document to be processed, and extracts a string of characters located at the position corresponding to the location information contained in the document information as the item name and item content corresponding to the item to be extracted from the document to be processed.(5) The information processing device according to any one of (1) to (4), characterized in that it has a correction means for receiving correction input for the extracted document information and correcting the document information in accordance with the correction input. (6) The information processing device according to any one of (1) to (5), characterized in that the document information stores the item contents corresponding to the items to be extracted in a list structure in which multiple item contents can be associated with one item. (7) The information processing device according to (4), characterized in that the information extraction means extracts strings that exist at the corresponding position if the center point of a rectangle calculated based on the position information of a string contained in the document to be processed is within a rectangle based on the position information contained in the document information obtained by searching. (8) The information processing device according to (4), characterized in that the information extraction means extracts strings that exist at the corresponding position if the distance between the center point of a rectangle calculated based on the position information of a string contained in the document to be processed and the center point of a rectangle calculated based on the position information contained in the document information obtained by searching is less than or equal to a predetermined threshold. (9) The information processing apparatus according to (4), wherein the information extraction means extracts strings that exist at the corresponding positions, wherein the degree of overlap between a rectangle based on the positional information of a string contained in the document to be processed and a rectangle based on the positional information of the document information obtained by searching is greater than or equal to a predetermined threshold.(10) An information processing method performed by an information processing device, comprising: a storage step of storing in a storage unit as document information, associated with the item to be extracted, an item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which are extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed. (11) A program (computer program product) for causing the computer of an information processing device to execute the following steps: a storage step of storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which have been extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.(12) A computer-readable recording medium characterized by recording a program for causing the computer of an information processing device to execute the following steps: a storage step of storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which have been extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

[0058] According to the present invention, it is possible to accurately extract the item name and item content corresponding to the target item from the image data of a document.

Claims

1. An information processing device comprising: a storage means for storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, extracted from documents processed in the past based on the item to be extracted; a search means for searching for the document information of a previously processed document corresponding to the document to be processed from the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction means for inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

2. The information processing apparatus according to claim 1, wherein the search means compares the item name and location information of the item name contained in the document information stored in the storage unit with the string and location information of the string contained in the document to be processed, and searches for the document information of the document corresponding to the document to be processed.

3. The information extraction means, if the item to be extracted is an item that may contain multiple items, uses the large-scale language model to extract the item name and item content corresponding to the item to be extracted from the document to be processed, and if the item to be extracted is an item that contains only one item content, extracts the item name and item content corresponding to the item to be extracted from the document to be processed based on the location information contained in the document information obtained by the search and the location information in the document to be processed, characterized in that the information processing device according to claim 1 or 2.

4. The information processing apparatus according to claim 3, wherein if the item to be extracted is an item that contains only one item content, the information extraction means performs a comparison using the location information contained in the document information obtained by the search and the location information in the document to be processed, and extracts the string present at the location corresponding to the location information contained in the document information from the document to be processed as the item name and item content corresponding to the item to be extracted.

5. The information processing device according to any one of claims 1 to 4, characterized in that it has a correction means for receiving correction input for the extracted document information and correcting the document information in accordance with the correction input.

6. The information processing device according to any one of claims 1 to 5, characterized in that the document information is stored in a list structure in which the item contents corresponding to the items to be extracted can be associated with a single item.

7. The information processing apparatus according to claim 4, characterized in that the information extraction means extracts strings that exist within the rectangle based on the position information contained in the document information obtained by the search, where the center point of the rectangle calculated based on the position information of the strings contained in the document to be processed is the corresponding string.

8. The information processing apparatus according to claim 4, characterized in that the information extraction means extracts strings as strings present at the corresponding positions, wherein the distance between the center point of a rectangle calculated based on the positional information of strings contained in the document to be processed and the center point of a rectangle calculated based on the positional information contained in the document information obtained by searching is less than or equal to a predetermined threshold.

9. The information processing apparatus according to claim 4, characterized in that the information extraction means extracts strings as strings present at the corresponding positions, wherein the degree of overlap between a rectangle based on the positional information of a string contained in the document to be processed and a rectangle based on the positional information of the document information obtained by searching is greater than or equal to a predetermined threshold.

10. An information processing method executed by an information processing device, comprising: a storage step of storing in a storage unit as document information, associated with the item to be extracted, an item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which have been extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from among the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

11. A program for causing the computer of an information processing device to execute the following steps: a storage step of storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which have been extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

12. A computer-readable recording medium characterized by recording a program for causing the computer of an information processing device to execute the following steps: a storage step of storing in a storage unit as document information, associated with the item to be extracted, the item name and item content corresponding to the item to be extracted, and position information indicating the respective locations of the item name and item content, which have been extracted from documents processed in the past based on the item to be extracted; a search step of searching for the document information of a previously processed document corresponding to the document to be processed from the document information stored in the storage unit, based on a string contained in the document to be processed and position information indicating the location of the string; and an information extraction step of inputting a prompt containing the document information obtained by the search into a large-scale language model, performing inference by the large-scale language model, and extracting the item name and item content corresponding to the item to be extracted from the document to be processed.

Images