Proofreading device, document recognition system, proofreading method, and program

Abstract

This proofreading device comprises: an acquisition unit that acquires document data including a recognition result obtained by executing optical character recognition on a document including a table; a table extraction unit that extracts table data including a recognition result of the table from the document data; an explanation extraction unit that extracts explanation information for the table from the document data; a proofreading instruction unit that instructs proofreading of the table data by inputting the table data and the explanation information to a machine learning model; and an output unit that outputs proofreading result data including a proofreading result obtained by proofreading the table data by the machine learning model.

Description

Calibration Device, Document Recognition System, Calibration Method, and Program

[0001] The present disclosure relates to a calibration device, a document recognition system, a calibration method, and a program.

[0002] Techniques for calibrating recognition results by optical character recognition (OCR) are known. For example, in Patent Document 1, characters included in a manuscript image are recognized and converted into text data, spelling mistake words in the text data are detected using dictionary data, and when a spelling mistake word is detected in the text data, candidate words similar to the spelling mistake word are displayed, and an image reading device that replaces the spelling mistake word in the text data with the candidate word selected by the user is disclosed.

[0003] Japanese Patent Application Laid-Open No. 2024-60455

[0004] However, in the prior art, the recognition results of tables included in a document cannot be calibrated. For example, the prior art calibrates a document based on a dictionary or context, but the data inside a table is often numerical and has almost no context.

[0005] One aspect of the present disclosure aims to calibrate a table included in a document.

[0006] The present disclosure includes the following configurations.

[0007] <1> An acquisition unit configured to acquire document data including recognition results obtained by performing optical character recognition on a document including a table, a table extraction unit configured to extract table data including the recognition results of the table from the document data, an explanation extraction unit configured to extract the explanation information of the table from the document data, a calibration instruction unit configured to instruct calibration of the table data by inputting the table data and the explanation information into a machine learning model, and an output unit configured to output calibration result data including a calibration result obtained by calibrating the table data by the machine learning model. A calibration device comprising.

[0008] <2> The calibration apparatus according to <1> above, wherein the explanation extraction unit is configured to extract the explanation information from the document data based on the title of the table.

[0009] <3> The proofreading device according to <2> above, further comprising a text splitting unit configured to split text data contained in the document data into a plurality of substrings, wherein the explanation extraction unit is configured to extract the explanation information based on the density of occurrence of the title in each of the plurality of substrings and the distance between each of the plurality of substrings and the table.

[0010] <4> The calibration apparatus according to <3> above, wherein the explanation extraction unit is configured to extract the explanation information based on a score weighted by the occurrence density and the distance.

[0011] <5> The calibration device described in <4> above, wherein the score is weighted based on a second machine learning model that has learned the relationship between tables and explanatory information in multiple documents, including tables.

[0012] <6> The machine learning model is a large-scale language model, the calibration device according to any of <1> to <5> above.

[0013] <7> The calibration device according to <6> above, wherein the calibration instruction unit is configured to input instruction information to the machine learning model indicating that the structure of the table should not be changed.

[0014] <8> The calibration device according to <6> or <7> above, wherein the calibration instruction unit is configured to input instruction information to the machine learning model indicating that blank cells should not be calibrated.

[0015] <9> The calibration device according to any one of <6> to <8> above, wherein the calibration instruction unit is configured to input instruction information to the machine learning model that instructs the machine learning model to duplicate the value of a second cell adjacent to the first cell into a first cell containing a predetermined abbreviation.

[0016] <10> A document recognition system comprising a recognition device and a proofreading device, wherein the recognition device comprises: a recognition unit configured to perform optical character recognition on a document including a table; and an output unit configured to output document data including the recognition result of the optical character recognition; and the proofreading device comprises: a table extraction unit configured to extract table data including the recognition result of the table from the document data; an explanation extraction unit configured to extract explanation information of the table from the document data; a proofreading instruction unit configured to instruct a machine learning model to proofread the table data by inputting the table data and the explanation information; and an output unit configured to output proofreading result data including the proofreading result obtained when the machine learning model proofreads the table data.

[0017] <11> A proofreading method comprising: a procedure for a computer to acquire document data including recognition results obtained by performing optical character recognition on a document containing a table; a procedure for extracting table data including the recognition results of the table from the document data; a procedure for extracting explanatory information of the table from the document data; a procedure for inputting the table data and the explanatory information into a machine learning model to instruct the machine learning model to proofread the table data; and a procedure for outputting proofreading result data including the proofreading results obtained by the machine learning model proofreading the table data.

[0018] <12> A program for causing a computer to perform the following steps: a procedure for acquiring document data including recognition results obtained by performing optical character recognition on a document containing a table; a procedure for extracting table data including the recognition results of the table from the document data; a procedure for extracting explanatory information of the table from the document data; a procedure for instructing a machine learning model to proofread the table data by inputting the table data and the explanatory information; and a procedure for outputting proofreading result data including the proofreading results obtained by the machine learning model proofreading the table data.

[0019] According to one aspect of this disclosure, tables included in a document can be corrected.

[0020] Figure 1 is a block diagram showing an example of the overall configuration of a document recognition system. Figure 2 is a block diagram showing an example of a computer. Figure 3 is a block diagram showing an example of the functional configuration of a document recognition system. Figure 4 is a diagram showing an example of document data. Figure 5 is the first diagram showing an example of a system prompt. Figure 6 is the second diagram showing an example of a system prompt. Figure 7 is the third diagram showing an example of a system prompt. Figure 8 is the fourth diagram showing an example of a system prompt. Figure 9 is the fifth diagram showing an example of a system prompt. Figure 10 is the sixth diagram showing an example of a system prompt. Figure 11 is a diagram showing an example of an assistant prompt. Figure 12 is a diagram showing an example of a user prompt. Figure 13 is a flowchart showing an example of a document recognition method. Figure 14 is a diagram showing an example of proofreading result data.

[0021] Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0022] [Embodiment] One embodiment of the present disclosure is an example of an information processing system that recognizes information contained in a document. Hereinafter, the information processing system according to this embodiment will be referred to as the "document recognition system". In this embodiment, the document recognition system performs optical character recognition on a document written on paper. The document recognition system has a function to correct the recognition results obtained by optical character recognition.

[0023] In this embodiment, the document may include a table. The table may be at least partially handwritten. For example, the table may have printed lines and data, or the table may have printed lines with handwritten data, or the lines and data may be handwritten. The data inside the table may include numbers, letters, symbols, or a combination thereof. For example, the data inside the table may include a string of characters that includes a measurement value and a unit.

[0024] This embodiment aims to proofread tables contained in a document. To this end, this embodiment extracts table data containing the recognition results of tables from document data containing recognition results obtained by performing optical character recognition on a document containing tables, extracts explanatory information about the tables from the document data, and instructs a machine learning model to proofread the table data by inputting the table data and explanatory information.

[0025] In one aspect, according to this embodiment, tables contained in a document can be corrected because the machine learning model is instructed to correct the table data. In another aspect, according to this embodiment, the machine learning model can correct the table data while referring to the table's descriptive information, so documents containing tables can be recognized with high accuracy.

[0026] <Overall Configuration> The overall configuration of the document recognition system according to this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the overall configuration of the document recognition system.

[0027] As shown in Figure 1, the document recognition system 1000 includes a recognition device 10, a language processing device 20, and a proofreading device 30. The recognition device 10, the language processing device 20, and the proofreading device 30 may be connected to each other via a communication network N to enable data communication. The recognition device 10 may not be connected to the communication network N and may be connected to the proofreading device 30 on a one-to-one basis via a different communication path than the communication network N. The communication network N may be, for example, a LAN (Local Area Network), a VPN (Virtual Private Network), or the Internet.

[0028] The recognition device 10 is an example of an electronic device that recognizes documents. The recognition device 10 may be a scanner, for example. The recognition device 10 is not limited to a scanner, and other examples include a copier, a facsimile machine, a digital multifunction printer (MFP; Multifunction Peripheral / Product / Printer), or a personal computer connected to a scanner. The recognition device 10 may be equipped with a control device that controls the operation of the recognition device 10. The control device may be a computer such as a microcomputer, for example.

[0029] The recognition device 10 may perform optical character recognition on a document. The recognition device 10 may capture an image of a document printed or handwritten on paper and perform optical character recognition on the image of the document. The recognition device 10 may perform optical character recognition on electronic data that includes an image of a document. The electronic data that includes an image of a document may be electronic data in any file format, for example, electronic data written in PDF (Portable Document Format) format.

[0030] The recognition device 10 may output electronic data that includes the recognition result obtained by performing optical character recognition on a document. Hereinafter, the electronic data that includes the recognition result of optical character recognition will be referred to as "document data".

[0031] The language processing device 20 is an example of an information processing device that performs a predetermined language processing task. The language processing device 20 may be a computer such as a personal computer, workstation, or server.

[0032] The language processing device 20 may have a pre-trained machine learning model. The language processing device 20 may perform a predetermined language processing task based on the pre-trained machine learning model. The language processing device 20 may perform a task of proofreading a document.

[0033] Machine learning models may be pre-trained to perform various language processing tasks. For example, a machine learning model could be a generative model such as a Large Language Model (LLM).

[0034] A machine learning model may be tuned to perform a specific language processing task. For example, a machine learning model may be further trained to accurately proofread documents. The method of further training may include, for example, transfer learning or fine-tuning.

[0035] The proofreading device 30 is an example of an information processing device for proofreading documents. The proofreading device 30 may be a computer such as a personal computer, workstation, or server.

[0036] The calibration device 30 may calibrate the recognition results obtained by the recognition device 10 when it recognizes a document. The calibration device 30 may calibrate the recognition results of a document that includes a table. The calibration device 30 may acquire document data from the recognition device 10 and calibrate at least a portion of the recognition results contained in the document data.

[0037] The calibration device 30 may instruct the language processing device 20 to calibrate the recognition result. The calibration device 30 may obtain the configured recognition result from the language processing device 20. The calibration device 30 may output electronic data containing the calibrated recognition result. Hereinafter, the electronic data containing the calibrated recognition result will be referred to as "calibration result data".

[0038] The overall configuration of the document recognition system 1000 shown in Figure 1 is just one example, and various system configurations are possible depending on the application and purpose. For example, one or more of the recognition device 10, language processing device 20, and proofreading device 30 may be included in multiple units of the document recognition system 1000. For example, the document recognition system 1000 may include terminal devices that perform data input and output. For example, the language processing device 20 or the proofreading device 30 may be implemented by multiple computers, or as a cloud computing service. For example, the document recognition system 1000 may be implemented by a standalone computer, or by a client-server system. The classification of devices such as the recognition device 10, language processing device 20, and proofreading device 30 shown in Figure 1 is just one example.

[0039] <Hardware Configuration> The hardware configuration of the document recognition system 1000 will be explained with reference to Figure 2. The control device that controls the recognition device 10, the language processing device 20, and the proofreading device 30 are implemented, for example, by a computer. Figure 2 is a block diagram showing an example of the computer's hardware configuration.

[0040] As shown in Figure 2, the computer 500 includes a CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, HDD (Hard Disk Drive) 504, input device 505, display device 506, communication interface 507, and external interface 508. The CPU 501, ROM 502, and RAM 503 form what is known as a computer. Each piece of hardware in the computer 500 is interconnected via a bus line 509. The input device 505 and display device 506 may also be connected to the computer 500 via the external interface 508 for use.

[0041] The CPU 501 is a computing device that reads programs and data from a storage device such as ROM 502 or HDD 504 onto RAM 503 and executes processing to realize the overall control and functions of the computer 500. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0042] ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. ROM 502 functions as the main memory, storing various programs and data necessary for the CPU 501 to execute the various programs installed on HDD 504. Specifically, ROM 502 stores boot programs such as BIOS (Basic Input Output System) and EFI (Extensible Firmware Interface) that are executed when the computer 500 starts up, as well as data such as OS (Operating System) settings and network settings.

[0043] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a DRAM (Dynamic Random Access Memory), a SRAM (Static Random Access Memory), or the like. The RAM 503 provides a work area in which various programs installed in the HDD 504 are developed when executed by the CPU 501.

[0044] The HDD 504 is an example of a non-volatile storage device that stores programs and data. Programs and data stored in the HDD 504 include an OS, which is basic software for controlling the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device (for example, an SSD: Solid State Drive or the like) that uses a flash memory as a storage medium instead of the HDD 504.

[0045] The input device 505 is a touch panel, operation keys or buttons, a keyboard or a mouse, a microphone for inputting audio data such as voice, etc., that are used by the user to input various signals.

[0046] The display device 506 is composed of a display such as a liquid crystal or an organic EL (Electro-Luminescence) that displays a screen, a speaker that outputs audio data such as voice, etc.

[0047] The communication I / F 507 is an interface for connecting to a communication network and enabling the computer 500 to perform data communication.

[0048] The external I / F 508 is an interface with an external device. External devices include a drive device 510 and the like.

[0049] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 511 may also include semiconductor memory that records information electrically, such as ROMs and flash memory. This allows the computer 500 to read and / or write to the recording medium 511 via the external interface 508.

[0050] The various programs to be installed on the HDD 504 are installed, for example, when the distributed recording medium 511 is set in a drive device 510 connected to an external I / F 508, and the various programs recorded on the recording medium 511 are read by the drive device 510. Alternatively, the various programs to be installed on the HDD 504 may be installed by downloading them via the communication I / F 507 from the communication network N or another network different from the communication network N.

[0051] <Functional Configuration> The functional configuration of the document recognition system 1000 will be explained with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration of the document recognition system.

[0052] <<Recognition Device>> As shown in Figure 3, the recognition device 10 comprises an imaging unit 110, a recognition unit 120, and an output unit 130. The recognition device 10 functions as the imaging unit 110, the recognition unit 120, and the output unit 130 when a recognition program pre-installed on the control device is executed.

[0053] For example, the imaging unit 110, the recognition unit 120, and the output unit 130 are realized by a process in which a program loaded from the HDD 504 shown in Figure 2 onto the RAM 503 is executed by the CPU 501.

[0054] The imaging unit 110 captures images of documents written on paper. The imaging unit 110 may also capture images of documents that include tables. The imaging unit 110 may also capture images of each page of a document that includes multiple pages. The imaging unit 110 may also generate image data that shows the images of the documents.

[0055] The recognition unit 120 performs optical character recognition. The recognition unit 120 may perform optical character recognition on an image of a document that has been captured. The recognition unit 120 may perform optical character recognition on an image of a document that includes a table. The recognition unit 120 may perform optical character recognition on an image shown in the image data generated by the imaging unit 110. The recognition unit 120 may perform optical character recognition on an image of a document that includes multiple pages, page by page.

[0056] The output unit 130 outputs document data. The document data may include an image of the document and the recognition result of the document. The recognition result may be the result of optical character recognition performed on the image by the recognition unit 120. The document data may also be data in which the recognition result is embedded in the image of the document. The recognition result may include one or more text data recognized from the image of the document and information indicating the location where the text data was recognized. As an example, the recognition result may include information indicating a region in which one or more recognized characters are written, identification information that identifies the region, and one or more characters recognized from the region.

[0057] Figure 4 shows an example of document data. Figure 4 shows document data that includes an image of a table contained in the document and the recognition result of that table. As shown in Figure 4, the document data may also include an image of the document. The document data may show rectangles (filled areas) indicating the areas where characters have been recognized. In addition, identification information (dotted line) that identifies the area and the recognized string of characters (dashed line) may be embedded in the vicinity of the areas where characters have been recognized. Note that in Figure 4, only some of the identification information and string of characters are shown with borders for visibility.

[0058] As shown in Figure 4, a table included in a document may contain multiple cells separated by arbitrary lines. Each cell in the table may contain data appropriate to the purpose of the table. For example, in a table showing experimental results, cells may contain only numerical values, or numerical values ​​combined with units. A table title may also be included near the table. The table title may include the table number.

[0059] ≪Calibration Device≫ Let's return to Figure 3 for explanation. As shown in Figure 3, the calibration device 30 comprises an acquisition unit 310, a table extraction unit 320, a text extraction unit 330, a text splitting unit 340, an explanation extraction unit 350, a calibration instruction unit 360, and an output unit 370. The calibration device 30 functions as the acquisition unit 310, table extraction unit 320, text extraction unit 330, text splitting unit 340, explanation extraction unit 350, calibration instruction unit 360, and output unit 370 when a pre-installed calibration program is executed.

[0060] For example, the acquisition unit 310, table extraction unit 320, text extraction unit 330, text splitting unit 340, explanation extraction unit 350, proofreading instruction unit 360, and output unit 370 are realized by a process in which a program, which is loaded from the HDD 504 shown in Figure 2 onto the RAM 503, is executed by the CPU 501.

[0061] The acquisition unit 310 acquires document data. The acquisition unit 310 may acquire document data output by the recognition device 10. The acquisition unit 310 may acquire document data received from the recognition device 10. The acquisition unit 310 may receive document data from a terminal device that acquired document data output by the recognition device 10. The acquisition unit 310 may accept input of document data via the input device 505 of the calibration device 30.

[0062] The table extraction unit 320 extracts table data from document data. Table data is electronic data that includes recognition results obtained by performing optical character recognition on the table. The table extraction unit 320 may extract table data corresponding to tables contained in the document. The table extraction unit 320 may extract multiple table data corresponding to multiple tables contained in the document.

[0063] Table data may include the recognition results of the data inside the table. Table data may include the recognition results of the string indicating the table title. Table data may include multiple recognition results recognized in each of the multiple cells contained in the table. Table data may also be data in which one or more characters recognized in each of the multiple cells are separated by a predetermined symbol. The predetermined symbol may be, for example, a tab, space, comma, pipe, newline, etc. Table data may also be data structured in a predetermined format. The predetermined format may be, for example, Markdown, HTML (Hyper Text Markup Language), JSON (JavaScript Object Notification), etc.

[0064] The table extraction unit 320 may recognize areas in a document where tables are written. The table extraction unit 320 may recognize areas where tables are written by recognizing the table lines or borders from the document image included in the document data. As an example, the table extraction unit 320 may recognize areas where tables are written based on image recognition or edge detection. The table extraction unit 320 may extract the recognition results embedded within the areas where tables are written as table data.

[0065] The text extraction unit 330 extracts text data from the document data. The text extraction unit 330 may also extract text data from the document data that indicates the recognition results recognized in areas other than tables. The text extraction unit 330 may also extract as text data the recognition results embedded in the document data, excluding the table data extracted by the table extraction unit 320. The text extraction unit 330 may also extract a single piece of text data from the entire document, including multiple pages.

[0066] The text splitting unit 340 divides the text data into multiple substrings. The text splitting unit 340 may also divide the text data into multiple substrings in predetermined processing units. The predetermined processing unit may be determined, for example, based on the number of characters, words, sentences, lines, paragraphs, or chapters.

[0067] The explanation extraction unit 350 extracts explanatory information for tables from text data. The explanation extraction unit 350 may extract any of the multiple substrings divided by the text splitting unit 340 as explanatory information. The explanation extraction unit 350 may also extract explanatory information corresponding to the table data extracted by the table extraction unit 320. The explanation extraction unit 350 may also extract explanatory information corresponding to each of the multiple table data.

[0068] The description extraction unit 350 may extract descriptive information for a table based on the table title. The description extraction unit 350 may search for a substring containing descriptive information for a table among multiple substrings based on the table title. The description extraction unit 350 may search for multiple substrings using a fuzzy search with the table title as the search key. Since table titles may contain misrecognitions, a fuzzy search can prevent descriptive information from being missed during the search.

[0069] The explanation extraction unit 350 may obtain the table title from the table data. The explanation extraction unit 350 may obtain the table title based on the positional relationship between the area where the table is described and the recognition result. The explanation extraction unit 350 may identify the table title based on a predetermined rule based on the coordinates of the area where the table is described and the coordinates of the recognition result. The explanation extraction unit 350 may instruct the language processing unit 20 to obtain the table title from the table data.

[0070] The explanation extraction unit 350 may extract explanatory information for the table based on a predetermined score. The predetermined score may be a weighted value obtained by considering the frequency of occurrence of the table title in multiple substrings and the distance between the area in which each of the multiple substrings is described and the area in which the table is described. The explanation extraction unit 350 may calculate a score for each of the multiple substrings and identify the substring with the highest score as the substring in which the explanatory information for the table is described.

[0071] The score can also be expressed using formula (1) as an example.

[0072]

[0073] Here, r is the frequency of occurrence, l is the distance, and a and b are predetermined weights. The frequency of occurrence r is the value obtained by dividing the number of occurrences of the word in the title within the substring by the total number of characters in the document. The distance l is the distance between the substring area and the table area. Specifically, the distance l is the absolute value of the distance obtained by subtracting the top-left coordinate of the table area from the top-left coordinate of the substring area.

[0074] Weights a and b may be determined based on machine learning. Weights a and b may be determined based on a machine learning model that has learned the positional relationship between tables and explanatory information in multiple documents containing tables. The machine learning model may be a classification model such as a support vector machine, for example.

[0075] As an example, weights a and b may be defined as follows: First, a predetermined number of papers are collected (for example, around 100). The text in the papers is divided into predetermined number of lines (for example, around 3 lines). The coordinates of the table and the coordinates of each divided text are obtained. A correct label, expressed as a truth value, is assigned to each divided text. As an example, the correct label may be 1 if it includes the explanatory information of the table, and 0 if it does not. Next, a support vector machine is trained based on training data including frequency of occurrence, distance, and correct labels. Then, weights a and b are determined based on the trained support vector machine. Note that the trained support vector machine is an example of a second machine learning model.

[0076] The calibration instruction unit 360 calibrates the table data. The calibration instruction unit 360 may instruct the language processing unit 20 to calibrate the table data. The calibration instruction unit 360 may also instruct the table extraction unit 320 to calibrate the table data extracted by the table extraction unit 320. The calibration instruction unit 360 may also instruct each of the multiple table data extracted by the table extraction unit 320 to calibrate the said table data.

[0077] The calibration instruction unit 360 may generate input information for the machine learning model. The calibration instruction unit 360 may transmit the input information for the machine learning model to the language processing unit 20. The input information for the machine learning model may, for example, be natural language sentences, also called prompts. The input information for the machine learning model may include multiple prompts. The prompts may, for example, include at least one of a system prompt, an assistant prompt, or a user prompt.

[0078] The input information to the machine learning model may include tabular data and descriptive information for the tables. The input information to the machine learning model may include instructions for correcting the tabular data. The input information to the machine learning model may include instructions indicating conditions for correction. The input information to the machine learning model may include instructions showing examples of correction.

[0079] The proofreading conditions may include information indicating the proofreading criteria. The proofreading conditions may include conditions prohibiting processing other than proofreading. The proofreading conditions may include conditions not to change the structure of the table. The proofreading conditions may include conditions not to proofread blank cells. The proofreading conditions may include conditions to duplicate the values ​​of other cells adjacent to a cell containing a specified ellipsis. The proofreading conditions may include conditions to proofread easily misrecognized characters to correct characters.

[0080] The calibration instruction unit 360 may acquire calibrated table data. The calibration instruction unit 360 may receive calibrated table data from the language processing unit 20. The calibration instruction unit 360 may acquire calibrated table data output from the machine learning model when the language processing unit 20 inputs input information into the machine learning model.

[0081] The output unit 370 outputs calibration result data. The calibration result data may include calibration table data. The calibration table data may be table data acquired by the calibration instruction unit 360 as a result of a machine learning model calibration of the table data. The calibration result data may also be data in which calibration table data is embedded in the document image. The calibration result data may also be data in which the table data extracted by the table extraction unit 320 from the document data acquired by the acquisition unit 310 has been replaced with calibration table data.

[0082] The output unit 370 may display the calibration result data on the display device 506 of the calibration device 30. The output unit 370 may transmit the calibration result data to the recognition device 10. The output unit 370 may transmit the calibration result data to the terminal device that the acquisition unit 310 has received the document data from. The output unit 370 may store the data in a storage device such as the HDD 504 of the calibration device 30 or in an external storage device.

[0083] It should be noted that the functional configuration of the document recognition system 1000 shown in Figure 3 is just one example, and there are various functional configurations depending on the application and purpose. The division of processing units shown in Figure 3, such as the imaging unit 110, recognition unit 120, output unit 130, acquisition unit 310, table extraction unit 320, text extraction unit 330, text splitting unit 340, explanation extraction unit 350, proofreading instruction unit 360, and output unit 370, is just one example.

[0084] <<Prompts>> A prompt, which is an example of input information to a machine learning model, will be explained with reference to Figures 5 to 12. In this embodiment, an example of a prompt including a system prompt, an assistant prompt, and a user prompt will be described.

[0085] Figures 5 to 10 show examples of system prompts. Note that Figures 5 to 10 show a single system prompt divided into parts. That is, a single system prompt is formed by linking system prompts 600 to 650 shown in Figures 5 to 10.

[0086] As shown in FIG. 5, the system prompt 600 may include instruction information 601 for instructing the calibration of table data, and instruction information 602 to 604 indicating conditions regarding the calibration. The instruction information 601 includes information indicating the format of the table data (in the example of FIG. 5, Markdown). The instruction information 602 indicates a condition to correct characters that are easily misrecognized (in the example of FIG. 5, "0", "お", "や", "わ") to correct characters (in the example of FIG. 5, "O", "第", "7"). The instruction information 603 indicates a condition to prohibit processing other than calibration (in the example of FIG. 5, translation). The instruction information 604 is an example of information indicating the criteria for calibration.

[0087] As shown in FIG. 6, the system prompt 610 may include instruction information 611 to 615 indicating conditions regarding the calibration. The instruction information 611 is an example of information indicating the criteria for calibration. The instruction information 612 and 614 indicate conditions not to change the structure of the table (in the example of FIG. 6, prohibition of changing the number of rows and columns, prohibition of deleting rows or columns, prohibition of changing the order). The instruction information 613 and 615 indicate conditions not to calibrate blank cells (in the example of FIG. 6, prohibition of filling missing values, prohibition of operating on blank cells).

[0088] As shown in FIG. 7, the system prompt 620 may include instruction information indicating a calibration example. The system prompt 620 shows a calibration example that satisfies the condition of not calibrating blank cells. In the system prompt 620, a calibration example is shown in which the blank cell 621 included in the table data before calibration remains as the blank cell 622 in the table data after calibration.

[0089] As shown in FIG. 8, the system prompt 630 may include instruction information indicating a calibration example. The system prompt 630 shows a calibration example that satisfies the condition of not calibrating blank rows (that is, a plurality of consecutive blank cells in the horizontal direction). In the system prompt 630, a calibration example is shown in which the blank row 631 included in the table data before calibration remains as the blank row 632 in the table data after calibration.

[0090] As shown in Figure 9, the system prompt 640 may include instruction information showing a calibration example. The system prompt 640 shows a calibration example that satisfies the condition that the value of another cell adjacent to a cell containing a predetermined ellipsis is duplicated. This condition may also imply that if adjacent cells also contain ellipsis, the value of an even more adjacent cell is duplicated. The system prompt 640 shows a calibration example in which cells 641 to 643 containing ellipsis in the table data before calibration are replaced in the table data after calibration with cells 646 to 648, which are duplicates of the values ​​of vertically adjacent cells 644 and 645.

[0091] As shown in Figure 10, the system prompt 650 may include instruction information showing proofreading examples. The system prompt 650 shows proofreading examples that satisfy the condition of correcting easily misrecognized characters to the correct characters. The system prompt 650 shows a proofreading example in which cell 651, which misrecognized "g" (gram) as "9" in the table data before proofreading, is corrected to "g" in cell 652 of the table data after proofreading. Similarly, the system prompt 650 shows a proofreading example in which cell 653, which misrecognized "l" (liter) as "1" in the table data before proofreading, is corrected to "l" in cell 654 of the table data after proofreading.

[0092] Figure 11 shows an example of an assistant prompt. As shown in Figure 11, the assistant prompt 660 may include a placeholder 661 into which explanatory information for a table is embedded. The explanatory information extracted by the explanatory information extraction unit 350 is embedded in the placeholder 661.

[0093] Figure 12 shows an example of a user prompt. As shown in Figure 12, the user prompt 670 may include a placeholder 671 into which table data is embedded. The placeholder 671 is filled with table data extracted by the table extraction unit 320.

[0094] The input information for a machine learning model may consist of a single prompt. For example, the input information for a machine learning model may be a single prompt formed by concatenating a system prompt, an assistant prompt, and a user prompt in that order.

[0095] <Processing Procedure> The document recognition method performed by the document recognition system 1000 will be explained with reference to Figure 13. Figure 13 is a flowchart showing an example of a document recognition method.

[0096] In step S1, the imaging unit 110 of the recognition device 10 captures an image of a document written on paper. The document includes at least one table and text data including explanatory information for the table. The imaging unit 110 generates image data showing the image of the document including the table. The imaging unit 110 sends the image data to the recognition unit 120.

[0097] In step S2, the recognition unit 120 of the recognition device 10 receives image data from the imaging unit 110. The recognition unit 120 performs optical character recognition on the image of the document shown in the image data. The recognition unit 120 sends the recognition result obtained by optical character recognition to the output unit 130.

[0098] The output unit 130 receives the recognition result from the recognition unit 120 using optical character recognition. The output unit 130 embeds the recognition result into the document image. This generates document data. The output unit 130 transmits the document data to the proofreading device 30.

[0099] In step S3, the calibration device 30 receives document data from the recognition device 10. The acquisition unit 310 of the calibration device 30 acquires the document data received by the calibration device 30. The acquisition unit 310 sends the document data to the table extraction unit 320 and the text extraction unit 330.

[0100] In step S4, the table extraction unit 320 of the proofreading device 30 receives document data from the acquisition unit 310. The table extraction unit 320 recognizes the area where a table is written based on the image of the document contained in the document data. The table extraction unit 320 extracts the recognition results embedded within the area where the table is written from the recognition results contained in the document data. This generates table data. The table extraction unit 320 sends the table data to the text extraction unit 330 and the explanation extraction unit 350.

[0101] In step S5, the text extraction unit 330 of the proofreading device 30 receives document data from the acquisition unit 310. The text extraction unit 330 also receives table data from the table extraction unit 320. The text extraction unit 330 extracts text data from the document data. The text extraction unit 330 extracts recognition results from the recognition results embedded in the document data, excluding the table data. The text extraction unit 330 sends the text data to the text splitting unit 340.

[0102] In step S6, the text splitting unit 340 of the proofreading device 30 receives text data from the text extraction unit 330. The text splitting unit 340 splits the text data into multiple substrings. The text splitting unit 340 sends the multiple substrings to the explanation extraction unit 350.

[0103] In step S7, the explanation extraction unit 350 of the calibration device 30 receives table data from the table extraction unit 320. The explanation extraction unit 350 also receives multiple substrings from the text splitting unit 340. The explanation extraction unit 350 obtains the table title from the table data.

[0104] In step S8, the explanation extraction unit 350 of the calibration device 30 searches for a substring containing table explanation information among the multiple substrings received in step S7, based on the table title acquired in step S7. The explanation extraction unit 350 extracts the found substring as table explanation information. The explanation extraction unit 350 sends the table data and table explanation information to the calibration instruction unit 360.

[0105] In step S9, the calibration instruction unit 360 of the calibration device 30 receives table data and table explanation information from the explanation extraction unit 350. Based on the table data and table explanation information, the calibration instruction unit 360 generates input information for the machine learning model. For example, the calibration instruction unit 360 embeds the table data into the user prompt placeholder and the table explanation information into the assistant prompt placeholder. The calibration instruction unit 360 also reads pre-stored system prompts. The calibration instruction unit 360 transmits the input information for the machine learning model to the language processing unit 20.

[0106] The language processing unit 20 receives input information from the proofreading device 30 to the machine learning model. The language processing unit 20 inputs the input information to the machine learning model. The machine learning model proofreads the tabular data based on the instruction and explanation information contained in the input information and outputs the proofread tabular data. The language processing unit 20 sends the proofread tabular data output by the machine learning model to the proofreading device 30.

[0107] The calibration device 30 receives the calibrated table data from the language processing device 20. The calibration instruction unit 360 of the calibration device 30 acquires the calibrated table data received by the calibration device 30. The calibration instruction unit 360 sends the calibrated table data to the output unit 370.

[0108] The processes from step S7 to step S9 are performed for each of the table data extracted in step S4. In other words, when optical character recognition is performed on a document containing multiple tables, the processes from step S7 to step S9 are repeated for each table in the document.

[0109] In step S10, the output unit 370 of the calibration device 30 receives the calibrated table data from the calibration instruction unit 360. The output unit 370 replaces the table data extracted in step S4 from the document data acquired in step S3 with the calibrated table data acquired in step S9. This generates the calibration result data.

[0110] The output unit 370 outputs calibration result data. The output unit 370 may also display the calibration result data on the display device 506 of the calibration device 30. The output unit 370 may also transmit the calibration result data to the recognition device 10.

[0111] Figure 14 shows an example of proofreading result data. Figure 14(A) shows an example of document data including table data before proofreading. Figure 14(B) shows an example of proofreading result data including proofread table data.

[0112] Figure 14(A) includes cell 701, which was misidentified. Specifically, cell 701 contains "2.1 × 10 １３ The recognition result shows "2.1 × 10¹³", which is an error where the superscript indicating the exponent was mistakenly recognized as a regular number.

[0113] Figure 14(B) includes cell 702 with the misrecognition corrected. Specifically, cell 702 contains "2.1 × 10 １３ The system correctly recognizes the superscript indicating the exponent and adds a caret (^) to show "2.1 × 10^13" as the recognition result.

[0114] <Effects of the Embodiment> The calibration device 30 according to this embodiment extracts table data, which includes the recognition results of a table, from document data, which includes the recognition results of optical character recognition performed on a document containing a table; extracts explanatory information about the table from the document data; instructs a machine learning model to calibrate the table data by inputting the table data and explanatory information; and outputs calibration result data, which includes the calibration results obtained by the machine learning model.

[0115] In one aspect, according to this embodiment, tables included in a document can be corrected by instructing a machine learning model to correct the tabular data. In another aspect, according to this embodiment, tables included in a document can be corrected with high accuracy because the machine learning model corrects the tabular data while referring to the descriptive information of the table.

[0116] The calibration device 30 may extract explanatory information from the document data based on the table title. In one respect, according to this embodiment, since explanatory information is extracted based on the table title, appropriate explanatory information can be extracted from the document data.

[0117] The proofreading device 30 may divide the text data contained in the document data into multiple substrings. The proofreading device 30 may extract explanatory information based on the density of title occurrences in each of the multiple substrings and the distance between each of the multiple substrings and the table. In one aspect, according to this embodiment, since explanatory information is extracted based on the density of title occurrences and the distance to the table, explanatory information can be extracted from the text data with high accuracy.

[0118] The calibration device 30 may extract explanatory information based on a score weighted by occurrence density and distance. The calibration device 30 may weight the score based on a second machine learning model that has learned the relationship between tables and explanatory information in multiple documents, including tables. In one aspect, according to this embodiment, explanatory information can be extracted from text data with high accuracy based on the relationship between tables and explanatory information in multiple documents.

[0119] The machine learning model may be a large-scale language model. In one respect, according to this embodiment, the machine learning model can perform various language processing tasks with high accuracy, and therefore tables contained in a document can be corrected with high accuracy.

[0120] The calibration device 30 may input instruction information to the machine learning model that instructs it not to change the structure of the table. In one respect, according to this embodiment, it is possible to suppress the change in the meaning of the entire table due to a change in the structure of the table.

[0121] The calibration device 30 may input instruction information to the machine learning model indicating that blank cells should not be calibrated. In one aspect, according to this embodiment, it is possible to suppress the change in the meaning of the entire table due to the modification of blank cells.

[0122] The calibration device 30 may input instruction information to a machine learning model that instructs the machine learning model to duplicate the value of a second cell adjacent to a first cell containing a predetermined ellipsis. In one aspect, according to this embodiment, since cells containing ellipsis can be replaced with their original values, the tabular data can be calibrated to match the intentions of the document creator.

[0123] [Supplement] Each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as CPUs (Central Processing Units) or GPUs (Graphics Processing Units) implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each function described above.

[0124] While embodiments of the present disclosure have been described in detail above, the embodiments disclosed herein are illustrative and not restrictive in all respects. The embodiments can be modified and improved in various ways without departing from the scope and spirit of the appended claims. The features described in the above embodiments can be combined in any way that is not inconsistent with other configurations.

[0125] This application claims priority to Japanese Patent Application No. 2024-185818, filed with the Japan Patent Office on 22 October 2024, which is incorporated herein by reference to its entire contents.

[0126] 10: Recognition device 20: Language processing device 30: Proofreading device 310: Acquisition unit 320: Table extraction unit 330: Text extraction unit 340: Text splitting unit 350: Explanation extraction unit 360: Proofreading instruction unit 370: Output unit 1000: Document recognition system

Claims

1. A calibration device comprising: an acquisition unit configured to acquire document data including recognition results obtained by performing optical character recognition on a document containing a table; a table extraction unit configured to extract table data including the recognition results of the table from the document data; an explanation extraction unit configured to extract explanatory information of the table from the document data; a calibration instruction unit configured to instruct a machine learning model to calibrate the table data by inputting the table data and the explanatory information; and an output unit configured to output calibration result data including the calibration results obtained by the machine learning model calibrating the table data.

2. The calibration apparatus according to claim 1, wherein the explanation extraction unit is configured to extract the explanation information from the document data based on the title of the table.

3. The proofreading device according to claim 2, further comprising a text splitting unit configured to split text data contained in the document data into a plurality of substrings, wherein the explanation extraction unit is configured to extract the explanation information based on the density of occurrence of the title in each of the plurality of substrings and the distance between each of the plurality of substrings and the table.

4. The calibration apparatus according to claim 3, wherein the explanation extraction unit is configured to extract the explanation information based on a score weighted by the occurrence density and the distance.

5. The calibration apparatus according to claim 4, wherein the score is weighted based on a second machine learning model that has learned the relationship between tables and explanatory information in a plurality of documents including tables.

6. The calibration device according to any one of claims 1 to 5, wherein the machine learning model is a large-scale language model.

7. The calibration device according to claim 6, wherein the calibration instruction unit is configured to input instruction information to the machine learning model indicating that the structure of the table should not be changed.

8. The calibration device according to claim 6 or 7, wherein the calibration instruction unit is configured to input instruction information to the machine learning model indicating that blank cells should not be calibrated.

9. The calibration device according to any one of claims 6 to 8, wherein the calibration instruction unit is configured to input instruction information to the machine learning model that instructs the machine learning model to duplicate the value of a second cell adjacent to the first cell into a first cell containing a predetermined abbreviation.

10. A document recognition system comprising a recognition device and a proofreading device, wherein the recognition device comprises: a recognition unit configured to perform optical character recognition on a document including a table; and an output unit configured to output document data including the recognition results of the optical character recognition; and the proofreading device comprises: a table extraction unit configured to extract table data including the recognition results of the table from the document data; an explanation extraction unit configured to extract explanation information of the table from the document data; a proofreading instruction unit configured to instruct a machine learning model to proofread the table data by inputting the table data and the explanation information; and an output unit configured to output proofreading result data including the proofreading results obtained by the machine learning model proofreading the table data.

11. A proofreading method comprising: a procedure for a computer to acquire document data including recognition results obtained by performing optical character recognition on a document containing a table; a procedure for extracting table data including the recognition results of the table from the document data; a procedure for extracting explanatory information of the table from the document data; a procedure for inputting the table data and the explanatory information into a machine learning model to instruct it to proofread the table data; and a procedure for outputting proofreading result data including the proofreading results obtained by the machine learning model proofreading the table data.

12. A program for causing a computer to perform the following steps: a procedure for acquiring document data including recognition results obtained by performing optical character recognition on a document containing a table; a procedure for extracting table data including the recognition results of the table from the document data; a procedure for extracting explanatory information of the table from the document data; a procedure for instructing a machine learning model to proofread the table data by inputting the table data and the explanatory information; and a procedure for outputting proofreading result data including the proofreading results obtained by the machine learning model proofreading the table data.

Images