Character acquisition system, character acquisition device, character acquisition program, and system

Abstract

This character acquisition system includes: an MLLM 30c serving as a model in which a document image serving as an image of a document and descriptive text for the document image are input to thereby output characters included in the document image; and an application 30d for acquiring characters serving as an output of the MLLM 30c in S292 by inputting an acquisition object document image serving as a document image from which characters are to be acquired and the descriptive text to the MLLM 30c (S290). The descriptive text includes character position identification sentences serving as sentences for identifying the positions of characters as values for each field in the document image.

Description

Character acquisition system, character acquisition apparatus, character acquisition program, and system

[0001] The present invention relates to a character acquisition system, a character acquisition apparatus, a character acquisition program, and a system for acquiring characters from a document image serving as an image of a document.

[0002] Conventionally, there has been known a character acquisition system that acquires characters from a document image by OCR (Optical Character Recognition) (see, for example, Patent Document 1). A conventional character acquisition system acquires characters from a document image by using, for example, a template for specifying the positions of characters in the document image.

[0003] Japanese Unexamined Patent Application Publication No. 2022-113038

[0004] However, for a specific document image such as a document image received by facsimile or a document image read from an original by a scanner, the position of a character may deviate from the position specified by the template. When the position of a character in the document image deviates from the position specified by the template, if characters are acquired from the document image using the template, there is a problem that it may not be possible to accurately acquire characters from the document image.

[0005] Therefore, an object of the present invention is to provide a character acquisition system, a character acquisition apparatus, a character acquisition program, and a system capable of improving the possibility of accurately acquiring characters from a document image.

[0006] A character acquisition system according to one aspect of the present invention includes an MLLM serving as a model that outputs characters included in a document image when the document image serving as an image of a document and a descriptive text of the document image are input thereto, and a character acquisition unit that acquires the characters output from the MLLM by inputting an acquisition target document image that is the document image from which characters are to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs characters included in the document image based on the descriptive text including a character position specifying text that is text for specifying the position of the character as a value for each item in the document image.

[0007] With this configuration, the character acquisition system according to one aspect of the present invention acquires characters as output of the MLLM by inputting a document image to be acquired, which is the document image from which characters are to be acquired, and a descriptive text of the document image, which is a text for identifying the position of characters as values ​​for each item in the document image, into the MLLM. This improves the possibility of accurately acquiring characters from a document image.

[0008] A character acquisition device according to one aspect of the present invention comprises an MLLM as a model that outputs characters contained in a document image when a document image as an image of a document and a descriptive text of the document image are input, and a character acquisition unit that acquires the characters as the output of the MLLM by inputting the target document image as the document image to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs the characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image.

[0009] With this configuration, a character acquisition device according to one aspect of the present invention acquires characters as output of the MLLM by inputting a document image to be acquired, which is the document image from which characters are to be acquired, and a descriptive text of the document image, which is a text for identifying the position of characters as values ​​for each item in the document image, into the MLLM. This improves the possibility of accurately acquiring characters from a document image.

[0010] A character acquisition program according to one aspect of the present invention operates a computer as follows: a Multimodal Language Model (MLLM) that integrates multiple modalities to infer as a model that outputs characters contained in a document image when a document image as an image of a document and a descriptive text of the document image are input; a character acquisition unit that acquires the characters as an output of the MLLM by inputting a target document image as the document image to be acquired and the descriptive text to the MLLM; and the computer further operates so that the MLLM outputs characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image.

[0011] With this configuration, a computer executing a character acquisition program according to one aspect of the present invention can acquire characters as output of the MLLM by inputting a document image, which is the target document image from which characters are to be acquired, and a descriptive text of the document image, which is a text for identifying the position of characters as values ​​for each item in the document image, into the MLLM. This improves the possibility of accurately acquiring characters from a document image.

[0012] A system according to one aspect of the present invention comprises a document processing system for processing a document image as an image of a document, and a character acquisition system for acquiring characters contained in the document image, wherein the character acquisition system comprises an MLLM as a model that outputs characters contained in the document image when the document image and a descriptive text of the document image are input, and a character acquisition unit that acquires the characters as the output of the MLLM by inputting the acquisition target document image as the document image to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs the characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image, and when the document image is input, the document processing system transmits the input document image as the input document image to the character acquisition system, the character acquisition unit uses the input document image as the acquisition target document image, and the document processing system manages the characters acquired by the character acquisition unit.

[0013] With this configuration, a system according to one aspect of the present invention can improve convenience because, when a document image is input to the document processing system, the document processing system transmits the input document image as an input document image to the character acquisition system, the character acquisition system acquires characters from the input document image, and the document processing system manages the characters acquired from the input document image by the character acquisition system.

[0014] A character acquisition system, character acquisition device, character acquisition program, and system according to one aspect of the present invention can improve the possibility of accurately acquiring characters from document images.

[0015] This is a block diagram of an example system according to one embodiment of the present invention. This is a block diagram of an example document processing system shown in Figure 1 when it is configured with one computer. This is a diagram of an example image database shown in Figure 2. This is a diagram of an example collection database shown in Figure 2. This is a block diagram of an example character acquisition system shown in Figure 1 when it is configured with one computer. This is a diagram of an example vector database shown in Figure 5. This is a diagram of an example description database shown in Figure 5. This is a block diagram of an example user terminal shown in Figure 1 when it is configured with a PC. This is a block diagram of an example image reader shown in Figure 1 when it is configured with an MFP. This is a sequence diagram of the operation of the system shown in Figure 1 when vector information and description of a document image are registered in the character acquisition system by operation from the user terminal. This is a sequence diagram of the operation of the system shown in Figure 1 when vector information and description of a document image are registered in the character acquisition system by operation from the document processing system. This is a sequence diagram of the operation of the system shown in Figure 1 when information is acquired from a document image by operation from the user terminal. This is a sequence diagram of the operation of the system shown in Figure 1 when a document image is input to the document processing system via email. This is a sequence diagram of the operation of the system shown in Figure 1 when a document image is input to the document processing system via a hot folder. Figure 1 is a sequence diagram of the system's operation when a document image is input to the document processing system 20 via a web page. Figure 1 is a sequence diagram of the system's operation when a document image is input to the document processing system from an image reader. Figure 1 is a sequence diagram of the system's operation when a document image is newly stored in the image database.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0017] First, the configuration of a system according to one embodiment of the present invention will be described.

[0018] Figure 1 is a block diagram of an example of the system 10 according to this embodiment.

[0019] As shown in Figure 1, System 10 includes a document processing system 20 that processes document images (hereinafter referred to as "document images"). The document processing system 20 may consist of one computer, such as a PC (Personal Computer), or it may consist of multiple computers. The document processing system 20 may also be configured on the cloud.

[0020] System 10 includes a character acquisition system (an example of a character acquisition device) 30 as a multimodal RAG (Retrievable Augmented Generation) that acquires characters contained in a document image from the document image. The character acquisition system 30 includes a vector management unit 30a that manages a vector DB as a database for managing vector information of the document image, a description management unit 30b that manages a description DB as a database for managing the description text of the document image, an MLLM (Multimodal Large Language Model) 30c that integrates multiple modalities to infer as a model that outputs characters contained in a document image when the document image and description text are input, and an application 30d as an orchestration tool that manages the vector management unit 30a, the description management unit 30b, and the MLLM 30c. The character acquisition system 30 may be configured by a single computer such as a PC, or by multiple computers. For example, the character acquisition system 30 may separately include a computer that implements a vector management unit 30a and a vector database, a computer that implements a description management unit 30b and a description database, a computer that implements the MLLM 30c, and a computer that implements the application 30d. The character acquisition system 30 may be configured on the cloud. The application 30d comprises a character acquisition unit that acquires characters as output from the MLLM 30c by inputting the document image to be acquired and the description into the MLLM 30c.

[0021] System 10 includes a user terminal 40 used by users. System 10 may also include at least one other user terminal used by users, such as a PC, tablet, or smartphone.

[0022] System 10 includes an image reading device 50 that reads an image from a document. System 10 may also include at least one other image reading device that reads an image from a document besides the image reading device 50. The image reading device 50 may be, for example, a dedicated scanner or an MFP (Multifunction Peripheral).

[0023] The document processing system 20 and the application 30d of the character acquisition system 30 can communicate with each other via a network 11 such as a LAN (Local Area Network) or the Internet. The user terminal and the document processing system 20 can communicate with each other via the network 11. The user terminal and the application 30d of the character acquisition system 30 can communicate with each other via the network 11. The image reading device and the document processing system 20 can communicate with each other via the network 11.

[0024] Figure 2 is a block diagram of an example of a document processing system 20 that is configured with a single computer.

[0025] As shown in Figure 2, the document processing system 20 includes an operation unit 21, which is an operation device such as a keyboard or mouse into which various operations are input; a display unit 22, which is a display device such as an LCD (Liquid Crystal Display) that displays various information; a communication unit 23, which is a communication device that communicates with external devices via a network such as a LAN or the Internet, or directly via wired or wireless connection without going through a network; a storage unit 24, which is a non-volatile storage device such as a semiconductor memory or HDD (Hard Disk Drive) that stores various information; and a control unit 25 that controls the entire document processing system 20.

[0026] The storage unit 24 can store a document processing program 24a for processing document images. The document processing program 24a may be installed in the document processing system 20 during the manufacturing stage, or it may be additionally installed in the document processing system 20 from an external storage medium such as a USB (Universal Serial Bus) memory, or it may be additionally installed in the document processing system 20 from a network.

[0027] The storage unit 24 is capable of storing an image DB 24b, which serves as a database for managing document images.

[0028] Figure 3 shows an example of image DB24b.

[0029] The image database 24b shown in Figure 3 contains an image ID as identification information for each document image, along with the document image itself. Some information is omitted in the image database 24b shown in Figure 3.

[0030] Each document image stored in image DB24b is a single-page image in a specific file format, such as JPEG (Joint Photographic Experts Group). As shown in Figure 3, the documents represented by the document images stored in image DB24b can be documents in various formats.

[0031] As shown in Figure 2, the storage unit 24 can store a collection DB 24c, which serves as a database for managing files containing information collected from the character acquisition system 30.

[0032] Figure 4 shows an example of the collected DB24c.

[0033] The collection DB24c shown in Figure 4 contains the image ID and file name for each document image. Some information is omitted in the depiction of the collection DB24c shown in Figure 4.

[0034] Each file stored in the collected DB24c is a file in a specific file format, such as JSON (JavaScript Object Notation) format.

[0035] The control unit 25 shown in Figure 2 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores programs and various data, and a RAM (Random Access Memory) used as a working area for the CPU of the control unit 25. The CPU of the control unit 25 executes programs stored in the storage unit 24 or the ROM of the control unit 25.

[0036] The control unit 25 implements a document processing unit 25a that processes document images by executing a document processing program 24a.

[0037] Figure 5 is a block diagram of an example of a character acquisition system 30 configured with a single computer.

[0038] As shown in Figure 5, the character acquisition system 30 includes an operation unit 31, which is an operation device such as a keyboard or mouse into which various operations are input; a display unit 32, which is a display device such as an LCD that displays various information; a communication unit 33, which is a communication device that communicates with external devices via a network such as a LAN or the Internet, or directly via wired or wireless connection without going through a network; a storage unit 34, which is a non-volatile storage device such as a semiconductor memory or HDD that stores various information; and a control unit 35 that controls the entire character acquisition system 30.

[0039] The storage unit 34 can store a character acquisition program 34a for acquiring characters contained in a document image from the document image. The character acquisition program 34a may be installed in the character acquisition system 30 during the manufacturing stage, or it may be additionally installed in the character acquisition system 30 from an external storage medium such as a USB memory, or it may be additionally installed in the character acquisition system 30 from a network.

[0040] The storage unit 34 is capable of storing a vector DB 34b.

[0041] FIG. 6 is a diagram showing an example of the vector DB 34b.

[0042] The vector DB 34b shown in FIG. 6 includes, for each document image, an index serving as identification information for vector information of the document image, and the vector information of the document image. The vector DB 34b shown in FIG. 6 is depicted with some information omitted.

[0043] As shown in FIG. 2, the storage unit 34 is capable of storing a description DB 34c.

[0044] FIG. 7 is a diagram showing an example of the description DB 34c.

[0045] The description DB 34c shown in FIG. 7 includes, for each description of a document image, an index associated with the description of the document image, and the description of the document image. The description DB 34c shown in FIG. 7 is depicted with some information omitted.

[0046] The description shown in FIG. 7 is an example of a description for a document image of a purchase order.

[0047] The description may include a text for specifying the position of a character as a value for each item in a document image (hereinafter referred to as a "character position specifying text"). For example, the character position specifying text may indicate in which column of a table at which position in the document image the value of which item is described. In the description shown in FIG. 7, the text "The item is described in the second column of the table in the center" is a character position specifying text indicating that the value of "item", which is one of the input items required for order reception processing, is described in the second column of the table at the center position in the document image. Similarly, the description may include a character position specifying text for each input item required for order reception processing. The input items required for order reception processing include, for example, order receiver code, order receiver name, item, quantity, unit price, total amount, and the like.

[0048] The descriptive text may include a sentence for identifying items whose values are handwritten among items in a document image (hereinafter referred to as a "handwritten item identification sentence"). In the descriptive text shown in FIG. 7, the sentence "the quantity and total amount of the order content are handwritten" is a handwritten item identification sentence indicating that among the items in the document image, the items whose values are handwritten are "quantity" and "total amount".

[0049] The control unit 35 shown in FIG. 5 includes, for example, a CPU, a ROM storing programs and various types of data, and a RAM used as a memory serving as a work area for the CPU of the control unit 35. The CPU of the control unit 35 executes programs stored in the storage unit 34 or the ROM of the control unit 35.

[0050] The control unit 35 implements a vector management unit 30a, a description management unit 30b, an MLLM 30c, and an application 30d by executing a character acquisition program 34a.

[0051] FIG. 8 is a block diagram of an example of the user terminal 40 when configured by a PC.

[0052] As shown in FIG. 8, the user terminal 40 includes an operation unit 41 which is an operation device such as a keyboard or a mouse for inputting various operations, a display unit 42 which is a display device such as an LCD for displaying various information, a communication unit 43 which is a communication device that communicates with external devices via a network such as a LAN or the Internet, or directly by wire or wireless without going through a network, a storage unit 44 which is a non-volatile storage device such as a semiconductor memory or an HDD for storing various information, and a control unit 45 that controls the entire user terminal 40.

[0053] The storage unit 44 can store a web browser program 44a, which is a program for a web browser. The web browser program 44a may be installed on the user terminal 40 during the manufacturing stage, or it may be additionally installed on the user terminal 40 from an external storage medium such as a USB memory, or it may be additionally installed on the user terminal 40 from a network.

[0054] The storage unit 44 is capable of storing an email client program 44b, which is a program for an email client. The email client program 44b may, for example, be installed on the user terminal 40 during the manufacturing stage of the user terminal 40, or it may be additionally installed on the user terminal 40 from an external storage medium such as a USB memory, or it may be additionally installed on the user terminal 40 from a network.

[0055] The storage unit 44 is capable of storing document images 44c. The storage unit 44 is capable of storing at least one document image other than document images 44c. The document images stored in the storage unit 44 may be, for example, document images read from the original by the image reading device 50. The file format of the document images stored in the storage unit 44 can be various file formats, such as PDF (Portable Document Format).

[0056] The control unit 45 includes, for example, a CPU, a ROM that stores programs and various data, and RAM as memory used as a workspace for the CPU of the control unit 45. The CPU of the control unit 45 executes programs stored in the storage unit 44 or the ROM of the control unit 45.

[0057] The control unit 45 implements a web browser 45a by executing a web browser program 44a.

[0058] The control unit 45 implements the email client 45b by executing the email client program 44b.

[0059] Figure 9 is a block diagram of an example of an image reading device 50 that is composed of an MFP.

[0060] As shown in Figure 9, the image reading device 50 is a computer comprising: an operation unit 51 which is an operation device such as buttons into which various operations are input; a display unit 52 which is a display device such as an LCD which displays various information; a printer 53 which is a printing device which prints images onto a recording medium such as paper; a scanner 54 which is a reading device which reads images from an original document; a communication unit 55 which is a communication device which communicates with external devices via a network such as a LAN or the Internet, or directly by wired or wireless connection without going through a network; a fax communication unit 56 which is a fax device which communicates faxes with an external facsimile device (not shown) via a communication line such as a public telephone line; a storage unit 57 which is a non-volatile storage device such as a semiconductor memory or HDD which stores various information; and a control unit 58 which controls the entire image reading device 50.

[0061] The storage unit 57 is capable of storing an operation control program 57a for controlling the operation of the image reading device 50. The operation control program 57a may, for example, be installed in the image reading device 50 during the manufacturing stage, or it may be additionally installed in the image reading device 50 from an external storage medium such as a USB memory, or it may be additionally installed in the image reading device 50 from a network.

[0062] The storage unit 57 is capable of storing document images 57b. The storage unit 57 is capable of storing at least one other document image besides document image 57b. The document image stored in the storage unit 57 may be, for example, a document image read from an original by the scanner 54, or a document image received by the fax communication unit 56. The file format of the document image stored in the storage unit 57 can be various file formats, such as PDF format.

[0063] The control unit 58 includes, for example, a CPU, a ROM that stores programs and various data, and RAM as memory used as a workspace for the CPU of the control unit 58. The CPU of the control unit 58 executes programs stored in the storage unit 57 or the ROM of the control unit 58.

[0064] The control unit 58 implements an operation control unit 58a that controls the operation of the image reading device 50 by executing the operation control program 57a.

[0065] Next, we will describe the operation of system 10.

[0066] First, we will explain the operation of system 10 when vector information of a document image and explanatory text are registered in the character acquisition system 30 by an operation from the user terminal 40.

[0067] Figure 10 is a sequence diagram of the operation of system 10 when vector information of a document image and explanatory text are registered in the character acquisition system 30 by an operation from the user terminal 40.

[0068] The user can instruct the user terminal 40 via the operation unit 41 to access a web page (hereinafter referred to as the "information registration page") for registering vector information and descriptive text of document images with the character acquisition system 30. The information registration page is provided by the application 30d of the character acquisition system 30. When the user terminal 40 is instructed to access the information registration page, the web browser 45a of the user terminal 40 accesses the information registration page as shown in Figure 10 (S101).

[0069] When the application 30d of the character acquisition system 30 receives an access in S101, it sends the data from the information registration page to the web browser 45a (S102).

[0070] When the web browser 45a of the user terminal 40 receives the data transmitted in S102, it displays an information registration page on the display unit 42 of the user terminal 40 based on the received data (S103). The information registration page allows the user to input a document image and a description of the document image, or, after the document image and its description have been input, the user can instruct to register the vector information and description of the document image.

[0071] When the web browser 45a is instructed on the information registration page to register the vector information of the document image and the descriptive text (hereinafter referred to as "information registration instruction"), it sends an instruction to the application 30d of the character acquisition system 30 (S104). The information registration instruction in S104 includes the document image and descriptive text that were entered on the information registration page at the time the instruction to register the vector information of the document image and the descriptive text was given on the information registration page.

[0072] When the application 30d of the character acquisition system 30 receives the information registration instruction transmitted in S104 (hereinafter referred to as the "target information registration instruction" in the operation description shown in Figure 10), it generates vector information of the document image by performing embedding on the document image included in the target information registration instruction (S105).

[0073] When the processing in S105 is completed, application 30d requests the vector management unit 30a to register the vector information generated in S105 (S106).

[0074] When the vector management unit 30a receives a request in S106, it assigns an index to the vector information for which registration was requested in S106 (S107).

[0075] When the processing in S107 is completed, the vector management unit 30a registers the combination of the vector information requested for registration in S106 and the index assigned in S107 in the vector DB 34b (S108).

[0076] When the processing in S108 is completed, the vector management unit 30a notifies the application 30d of the index assigned in S107 (S109).

[0077] When application 30d receives notification of the index in S109, it requests the explanatory text management unit 30b to register the index notified in S109 and the explanatory text included in the target information registration instruction (S110).

[0078] When the description management unit 30b receives a request in S110, it registers the combination of index and description requested to be registered in S110 into the description DB 34c (S111).

[0079] When the processing in S111 is completed, the description management unit 30b notifies the application 30d that the registration of the index and description combination is complete (S112).

[0080] When application 30d is notified in S112 that the registration of the index and description combination is complete, it notifies the web browser 45a of the user terminal 40 that the registration of the document image vector information and description is complete (S113).

[0081] When the user terminal 40's web browser 45a is notified in S113 that the registration of the document image's vector information and explanatory text is complete, it displays the completion of the registration of the document image's vector information and explanatory text on the display unit 42 (S114).

[0082] Next, we will describe the operation of system 10 when vector information of a document image and explanatory text are registered in the character acquisition system 30 by an operation from the document processing system 20.

[0083] Figure 11 is a sequence diagram of the operation of system 10 when vector information of a document image and explanatory text are registered in the character acquisition system 30 by an operation from the document processing system 20.

[0084] The user can instruct the document processing system 20 to register a combination of vector information of a specific document image stored in the document processing system 20 and an explanatory text entered into the document processing system 20 via the operation unit 21 of the document processing system 20, or via the communication unit 23 of the document processing system 20 via a user terminal such as the user terminal 40, to the character acquisition system 30, as shown in Figure 11. When the document processing unit 25a of the document processing system 20 is instructed to register a combination of vector information of a specific document image stored in the document processing system 20 and an explanatory text to the character acquisition system 30, it sends an information registration instruction to the application 30d of the character acquisition system 30, which includes the document image specified by the user from among the document images stored in the image DB 24b and the explanatory text entered into the document processing system 20 by the user (S121).

[0085] When the application 30d of the character acquisition system 30 receives the information registration instruction transmitted in S121 (hereinafter referred to as the "target information registration instruction" in the operation description shown in Figure 11), it generates vector information of the document image by performing embedding on the document image included in the target information registration instruction (S122).

[0086] When the processing in S122 is completed, application 30d requests the vector management unit 30a to register the vector information generated in S122 (S123).

[0087] When the vector management unit 30a receives a request in S123, it assigns an index to the vector information for which registration was requested in S123 (S124).

[0088] When the processing in S124 is completed, the vector management unit 30a registers the combination of the vector information requested for registration in S123 and the index assigned in S124 in the vector DB 34b (S125).

[0089] When the processing in S125 is completed, the vector management unit 30a notifies the application 30d of the index assigned in S124 (S126).

[0090] When application 30d receives notification of the index in S126, it requests the explanatory text management unit 30b to register the index notified in S126 and the explanatory text included in the target information registration instruction (S127).

[0091] When the description management unit 30b receives a request in S127, it registers the combination of index and description requested to be registered in S127 in the description DB 34c (S128).

[0092] When the processing in S128 is completed, the description management unit 30b notifies the application 30d that the registration of the index and description combination is complete (S129).

[0093] When application 30d is notified in S129 that the registration of the combination of index and description has been completed, it notifies the document processing system 20 that the registration of the vector information of the document image and the description has been completed (S130).

[0094] When the document processing unit 25a of the document processing system 20 is notified in S130 that the registration of the vector information of the document image and the explanatory text has been completed, it displays the completion of the registration of the vector information of the document image and the explanatory text (S131). Here, if the document processing unit 25a has been instructed via the operation unit 21 of the document processing system 20 to register a combination of vector information of a specific document image stored in the document processing system 20 and an explanatory text with the character acquisition system 30, it displays the completion of the registration of the vector information of the document image and the explanatory text on the display unit 22 of the document processing system 20. If the document processing unit 25a has been instructed via the communication unit 23 of the document processing system 20 through a user terminal to register a combination of vector information of a specific document image stored in the document processing system 20 and an explanatory text with the character acquisition system 30, it displays the completion of the registration of the vector information of the document image and the explanatory text on the display unit of the user terminal.

[0095] Next, we will explain the operation of the system 10 when information is obtained from a document image through an operation from the user terminal 40.

[0096] Figure 12 is a sequence diagram of the operation of system 10 when information is acquired from a document image through an operation from the user terminal 40.

[0097] The user can instruct the user terminal 40 via the operation unit 41 of the user terminal 40 to access a web page for obtaining information from document images (hereinafter referred to as the "information acquisition page"). The information acquisition page is provided by the application 30d of the character acquisition system 30. When the user terminal 40 is instructed to access the information acquisition page, the web browser 45a of the user terminal 40 accesses the information acquisition page as shown in Figure 12 (S141).

[0098] When the application 30d of the character acquisition system 30 receives an access in S141, it sends the data of the information acquisition page to the web browser 45a (S142).

[0099] When the web browser 45a of the user terminal 40 receives the data transmitted in S142, it displays an information retrieval page on the display unit 42 of the user terminal 40 based on the received data (S143). The information retrieval page can be either a document image specified by the user, or a document image is specified, and then a prompt is entered by the user instructing the user to retrieve values ​​for specific items from this document image. For example, the prompt may include the sentence, "Please tell me the customer code, customer name, item, quantity, unit price, and total amount."

[0100] When a prompt is entered on the information retrieval page, the web browser 45a sends an instruction to the application 30d of the character acquisition system 30 to retrieve information from the document image specified on the information retrieval page (hereinafter referred to as the "specified image information retrieval instruction") (S144). The specified image information retrieval instruction in S144 includes the prompt that was entered on the information retrieval page and the document image that was specified on the information retrieval page at the time the prompt was entered.

[0101] When the application 30d of the character acquisition system 30 receives the specified image information acquisition instruction transmitted in S144 (hereinafter referred to as the "target specified image information acquisition instruction" in the operation description shown in Figure 12), it generates vector information of the document image by performing embedding on the document image included in the target specified image information acquisition instruction (S145).

[0102] When the processing in S145 is completed, application 30d requests the vector management unit 30a to provide the index corresponding to the vector information generated in S145 (S146).

[0103] When the vector management unit 30a receives the request in S146, it uses semantic search to identify the vector information most similar to the vector information requested in S146 from among the vector information stored in the vector DB 34b (S147).

[0104] When the processing in S147 is completed, the vector management unit 30a notifies the application 30d of the index corresponding to the vector information identified in S147 (S148).

[0105] When application 30d is notified of an index in S148, it requests the explanation management unit 30b to provide an explanation corresponding to the index notified in S148 (S149).

[0106] When the description management unit 30b receives the request in S149, it identifies the description stored in the description DB 34c that is associated with the index that is the target of the request in S149 (S150).

[0107] When the processing in S150 is completed, the description management unit 30b notifies the application 30d of the description identified in S150 (S151).

[0108] When application 30d receives notification of the explanatory text in S151, it adds the explanatory text notified in S151 to the prompt included in the instruction to acquire target specified image information (S152).

[0109] In S152, application 30d inputs a prompt with an added explanatory text and the document image included in the instruction to acquire target specified image information to MLLM 30c (S153).

[0110] When a prompt and a document image are input in S153, the MLLM30c extracts characters from the document image input in S153 for each item for which value acquisition is instructed in the prompt input in S153 (S154). Here, if the explanatory text included in the prompt input in S153 contains a character position identification sentence, the MLLM30c extracts characters by OCR (Optical Character Recognition) according to the position indicated in this character position identification sentence. Furthermore, if the explanatory text included in the prompt input in S153 contains a handwritten item identification sentence, the MLLM30c extracts characters as values ​​for the items indicated in this handwritten item identification sentence by handwriting OCR. The MLLM30c may have an OCR function as shown in this embodiment, or alternatively, it may access an external device and call up and use the OCR function provided in the external device.

[0111] When the processing in S154 is completed, MLLM30c outputs the characters for each item extracted in S154 to application 30d (S155).

[0112] When the processing in S155 is completed, application 30d notifies the web browser 45a of the user terminal 40 of the characters for each item, i.e., the values ​​for each item, that were output in S155 (S156).

[0113] When the value for each item is notified in S156, the web browser 45a of the user terminal 40 displays the value for each item notified in S156 on the display unit 42 (S157).

[0114] Next, we will describe the operation of system 10 when a document image is input to the document processing system 20 via email.

[0115] Figure 13 is a sequence diagram of the operation of system 10 when a document image is input to the document processing system 20 via email.

[0116] The user can instruct the user terminal 40 via the operation unit 41 of the user terminal 40 to send an email with a document image stored on the user terminal 40 attached to the email address of the document processing system 20. When the email client 45b of the user terminal 40 is instructed to send an email with a document image stored on the user terminal 40 attached to the email address of the document processing system 20, it sends an email with the document image specified by the user from among the document images stored in the storage unit 44 attached to the email address of the document processing system 20, as shown in Figure 13 (S201).

[0117] The document processing unit 25a of the document processing system 20 periodically checks whether there are any unreceived emails addressed to the document processing system 20's email address on the mail server. If such unreceived emails exist on the mail server, the document processing unit 25a receives the unreceived emails from the mail server (S202).

[0118] When the document processing unit 25a receives an email in S202, if the document image attached to the email received in S202 consists of multiple pages, it divides the document image attached to the email received in S202 into individual pages (S203).

[0119] If the document processing unit 25a receives an email in S202 and the attached document image does not consist of multiple pages, or if the processing in S203 is completed, it converts the file format of each page of the document image to a specific file format, such as JPEG (S204).

[0120] When the processing in S204 is completed, the document processing unit 25a assigns an image ID to the document image that was converted to a specific file format in S204 (S205).

[0121] When the processing in S205 is completed, the document processing unit 25a stores the combination of the image ID assigned to the document image in S205 and the document image converted to a specific file format in S204 in the image DB 24b (S206).

[0122] Next, we will describe the operation of system 10 when a document image is input to the document processing system 20 via a hot folder.

[0123] Figure 14 is a sequence diagram of the operation of system 10 when a document image is input to the document processing system 20 via a hot folder.

[0124] The user can instruct the user terminal 40 via the operation unit 41 of the user terminal 40 to send document images stored on the user terminal 40 to a hot folder for the document processing system 20. When the web browser 45a of the user terminal 40 is instructed to send document images stored on the user terminal 40 to a hot folder for the document processing system 20, it sends the document images specified by the user from among the document images stored in the storage unit 44 to the hot folder for the document processing system 20, as shown in Figure 14 (S221).

[0125] The document processing unit 25a of the document processing system 20 periodically checks whether a document image exists in the hot folder for the document processing system 20. If a document image exists in the hot folder for the document processing system 20, the document processing unit 25a retrieves the document image from the hot folder for the document processing system 20 (S222).

[0126] When the document processing unit 25a acquires a document image in S222, if the document image acquired in S222 consists of multiple pages, it divides the document image acquired in S222 into individual pages (S223).

[0127] If the document image acquired in S222 does not consist of multiple pages, or if the processing in S223 is completed, the document processing unit 25a converts the file format of each page of the document image to a specific file format, such as JPEG (S224).

[0128] When the processing in S224 is completed, the document processing unit 25a assigns an image ID to the document image that was converted to a specific file format in S224 (S225).

[0129] When the processing in S225 is completed, the document processing unit 25a stores the combination of the image ID assigned to the document image in S225 and the document image converted to a specific file format in S224 in the image DB 24b (S226).

[0130] Next, we will describe the operation of system 10 when a document image is input to the document processing system 20 via a web page.

[0131] Figure 15 is a sequence diagram of the operation of system 10 when a document image is input to the document processing system 20 via a web page.

[0132] The user can instruct the user terminal 40 to access a web page for uploading document images (hereinafter referred to as the "upload page") via the operation unit 41 of the user terminal 40. The upload page is provided by the document processing unit 25a of the document processing system 20. When the user terminal 40 is instructed to access the upload page, the web browser 45a of the user terminal 40 accesses the upload page as shown in Figure 15 (S241).

[0133] When the document processing unit 25a of the document processing system 20 receives an access request in S241, it sends the data of the upload page to the web browser 45a (S242).

[0134] When the web browser 45a of the user terminal 40 receives the data transmitted in S242, it displays an upload page on the display unit 42 of the user terminal 40 based on the received data (S243). The upload page allows the user to input a document image, or to input a document image and then instruct the user to upload the document image.

[0135] When the web browser 45a is instructed to upload a document image on the upload page, it sends the document image that was entered on the upload page at the time the upload was instructed to upload a document image to the document processing system 20 (S244).

[0136] When the document processing unit 25a of the document processing system 20 receives a document image transmitted by the user terminal 40 in S244, if the document image received from the user terminal 40 consists of multiple pages, it divides the document image received from the user terminal 40 into individual pages (S245).

[0137] If the document image received from the user terminal 40 does not consist of multiple pages, or if the processing in S245 is completed, the document processing unit 25a converts the file format of each page of the document image to a specific file format, such as JPEG (S246).

[0138] When the processing in S246 is completed, the document processing unit 25a assigns an image ID to the document image that was converted to a specific file format in S246 (S247).

[0139] When the processing in S247 is completed, the document processing unit 25a stores the combination of the image ID assigned to the document image in S247 and the document image converted to a specific file format in S246 in the image DB 24b (S248).

[0140] Next, we will describe the operation of the system 10 when a document image is input from the image reading device 50 to the document processing system 20.

[0141] Figure 16 is a sequence diagram of the operation of system 10 when a document image is input from the image reading device 50 to the document processing system 20.

[0142] The user can instruct the image reader 50 via the operation unit 51 of the image reader 50 to input a document image stored in the image reader 50 into the document processing system 20. When the operation control unit 58a of the image reader 50 is instructed to input a document image stored in the image reader 50 into the document processing system 20, it transmits the document image instructed by the user from among the document images stored in the storage unit 57 to the document processing system 20, as shown in Figure 16 (S261).

[0143] In S261, when the document processing unit 25a of the document processing system 20 receives a document image transmitted by the image reading device 50, if the document image received from the image reading device 50 consists of multiple pages, it divides the document image received from the image reading device 50 into individual pages (S262).

[0144] If the document image received from the image reading device 50 does not consist of multiple pages, or if the processing in S262 is completed, the document processing unit 25a converts the file format of each page of the document image to a specific file format, such as JPEG (S263).

[0145] When the processing in S263 is completed, the document processing unit 25a assigns an image ID to the document image that was converted to a specific file format in S263 (S264).

[0146] When the processing in S264 is completed, the document processing unit 25a stores the combination of the image ID assigned to the document image in S264 and the document image converted to a specific file format in S263 in the image DB 24b (S265).

[0147] Next, we will describe the operation of the system 10 when a new document image is stored in the image database 24b.

[0148] Figure 17 is a sequence diagram of the operation of system 10 when a new document image is stored in image DB 24b.

[0149] When the document processing unit 25a of the document processing system 20 newly stores a document image in the image database 24b, it sends an instruction to the application 30d of the character acquisition system 30 to retrieve information from the document image stored in the image database 24b (hereinafter referred to as the "stored image information retrieval instruction"), as shown in Figure 17 (S281). The stored image information retrieval instruction in S281 includes the document image newly stored in the image database 24b.

[0150] When the application 30d of the character acquisition system 30 receives the instruction to acquire stored image information transmitted in S281 (hereinafter referred to as the "target stored image information acquisition instruction" in the operation description shown in Figure 17), it generates vector information of the document image by performing embedding on the document image included in the target stored image information acquisition instruction (S282).

[0151] When the processing in S282 is completed, application 30d requests the vector management unit 30a to provide the index corresponding to the vector information generated in S282 (S283).

[0152] When the vector management unit 30a receives the request in S283, it uses semantic search to identify the vector information most similar to the vector information requested in S283 from among the vector information stored in the vector DB 34b (S284).

[0153] When the processing in S284 is completed, the vector management unit 30a notifies the application 30d of the index corresponding to the vector information identified in S284 (S285).

[0154] When application 30d receives notification of an index in S285, it requests the explanation management unit 30b to provide an explanation corresponding to the index notified in S285 (S286).

[0155] When the description management unit 30b receives the request in S286, it identifies the description stored in the description DB 34c that is associated with the index that is the target of the request in S286 (S287).

[0156] When the processing in S287 is completed, the description management unit 30b notifies the application 30d of the description identified in S287 (S288).

[0157] When application 30d receives notification of the explanatory text in S288, it generates a prompt (S289) based on the character position identification text of the explanatory text notified in S288, instructing the acquisition of values ​​for each item indicated in the character position identification text of the explanatory text notified in S288 from the document image included in the instruction to acquire target stored image information. Application 30d also includes the explanatory text notified in S288 itself in the prompt generated in S289.

[0158] Application 30d inputs the prompt generated in S289 and the document image included in the instruction to acquire target stored image information to MLLM 30c (S290).

[0159] When a prompt and a document image are input in S290, the MLLM30c extracts characters from the document image input in S290 for each item for which value acquisition is instructed in the prompt input in S290 (S291). Here, if the explanatory text included in the prompt input in S290 contains a character position identification sentence, the MLLM30c extracts characters by OCR according to the position indicated in this character position identification sentence. Furthermore, if the explanatory text included in the prompt input in S290 contains a handwritten item identification sentence, the MLLM30c extracts characters as the values ​​of the items indicated in this handwritten item identification sentence by handwriting OCR.

[0160] When the processing in S291 is completed, MLLM30c outputs the characters for each item extracted in S291 to application 30d (S292).

[0161] When the processing in S292 is completed, application 30d generates a file in a specific file format, such as JSON format, which contains the characters for each item, i.e., the values ​​for each item, that were output in S292 (S293).

[0162] When the processing in S292 is completed, application 30d sends the file generated in S293 to the document processing system 20 (S294).

[0163] When the document processing unit 25a of the document processing system 20 receives the file transmitted in S294, it stores the combination of the received file and the image ID associated with the document image included in the instruction to acquire target stored image information in the image database 24b in the collection database 24c (S295).

[0164] As explained above, the character acquisition system 30 acquires characters as the output of the MLLM 30c (S155, S292) by inputting a document image to be acquired as the target document image for character acquisition, and a descriptive text of the document image which includes a text for identifying the position of characters as values ​​for each item in the document image (S153, S290) into the MLLM 30c. Therefore, the possibility of accurately acquiring characters from a document image can be improved.

[0165] The character acquisition system 30 generates target vector information as vector information of the target document image by performing embedding on the target document image (S145, S282), identifies similar vector information as vector information most similar to the target vector information from among the vector information managed by the vector management unit 30a by semantic search (S147, S284), identifies related explanatory texts as explanatory texts associated with the similar vector information from among the explanatory texts managed by the explanatory text management unit 30b (S150, S287), and acquires characters as output of the MLLM 30c by inputting the target document image and related explanatory texts into the MLLM 30c (S153, S290). As a result, the possibility of inputting the target document image and appropriate related explanatory texts for this target document image into the MLLM 30c is improved, and the possibility of accurately acquiring characters from the document image is improved.

[0166] The character acquisition system 30 generates registered vector information as vector information of the registered document image by performing embedding on the registered document image, which is the document image to be registered as vector information (S105, S122), manages the registered vector information (S108, S125), and manages the explanatory text entered in association with the registered document image in association with the registered vector information (S111, S128). Therefore, the registration of vector information of a document image and the explanatory text associated with this vector information can be facilitated.

[0167] The character acquisition system 30 acquires characters as output from the MLLM 30c (S155, S292) by inputting the target document image and a descriptive text of the document image, which includes handwritten item identification text that identifies items in the document image whose values ​​are handwritten, into the MLLM 30c (S153, S290). As a result, the values ​​of the items indicated in the handwritten item identification text can be acquired by handwriting OCR, and the possibility of accurately acquiring characters from the document image can be improved.

[0168] When a document image is input to the document processing system 20 in S202, S222, S244, or S261, the document processing system 20 transmits the input document image as an input document image to the character acquisition system 30 (S281), the character acquisition system 30 acquires characters from the input document image (S292), and the document processing system 20 manages the characters acquired from the input document image by the character acquisition system 30 (S295), thus improving convenience.

Claims

1. A character acquisition system comprising: an MLLM that integrates multiple modalities to perform inference as a model that outputs characters contained in a document image when a document image as an image of a document and a descriptive text of the document image are input; and a character acquisition unit that acquires the characters as an output of the MLLM by inputting a target document image as the document image to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs the characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image.

2. A character acquisition system according to claim 1, comprising: a vector management unit for managing vector information of a document image; a description management unit for managing the description associated with the vector information, wherein the character acquisition unit generates target vector information as the vector information of the target document image by performing embedding on the target document image; the vector management unit identifies similar vector information as the vector information most similar to the target vector information among the vector information managed by the vector management unit itself by semantic search; the description management unit identifies related description as the description associated with the similar vector information among the description managed by the description management unit itself; and the character acquisition unit acquires the character as the output of the MLLM by inputting the target document image and the related description into the MLLM.

3. The character acquisition unit generates registered vector information as vector information of the registered document image by performing embedding on the registered document image as the document image to be registered as the target of registration of the vector information; the vector management unit manages the registered vector information; and the description management unit manages the description input in association with the registered document image in association with the registered vector information, the character acquisition system according to claim 2.

4. The character acquisition system according to claim 1, wherein the MLLM accepts input of an explanatory text which includes a handwritten item identification text as text for identifying the item in the document image whose value is handwritten.

5. A character acquisition device comprising: an MLLM that integrates multiple modalities to infer as a model that outputs characters contained in a document image when a document image as an image of a document and a descriptive text of the document image are input; and a character acquisition unit that acquires the characters as an output of the MLLM by inputting a target document image as the document image to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs the characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image.

6. A character acquisition program that operates a computer as a model that integrates multiple modalities to infer characters contained in a document image, by inputting a document image as an image of a document and a descriptive text for the document image; a character acquisition unit that acquires the characters as an output of the MLLM by inputting a target document image as the target document image for character acquisition and the descriptive text for the MLLM; and further operates the computer so that the MLLM outputs characters contained in the document image based on the descriptive text which includes a character position identification sentence as a sentence for identifying the position of the characters as values ​​for each item in the document image.

7. A system comprising: a document processing system for processing a document image as an image of a document; and a character acquisition system for acquiring characters contained in the document image, wherein the character acquisition system comprises: an MLLM that integrates multiple modalities to infer as a model that outputs characters contained in the document image when the document image and a descriptive text of the document image are input; a character acquisition unit that acquires the characters as an output of the MLLM by inputting the target document image as the document image to be acquired and the descriptive text to the MLLM, wherein the MLLM outputs the characters contained in the document image based on the descriptive text which includes a character position identification text as a sentence for identifying the position of the characters as values ​​for each item in the document image; the document processing system transmits the input document image as the input document image to the character acquisition system when the document image is input; the character acquisition unit uses the input document image as the target document image; and the document processing system manages the characters acquired by the character acquisition unit.

8. The document processing system according to claim 7, wherein the document processing system stores a file containing the values ​​for each item acquired by the character acquisition unit in a collection DB, and manages combinations of the file and the image ID associated with the document image.

Images