Image processing device and image processing method

Abstract

An entry object detection unit (12) detects, in a document image of a document, entry objects entered in the document. An entry information extraction unit (13) detects an entry object overlaid with a corrected object among the detected entry objects, and excludes the entry object overlaid with the corrected object, thereby extracting entry information corresponding to the detected entry objects.

Description

Image Processing Apparatus and Image Processing Method ,

[0008] ,

[0007] , ,

[0006] , ,

[0005] ,

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] An image processing apparatus detects non-character objects in a read image of a document having an area where a character string is entered or stamped, performs character recognition processing on character blocks included in the read image to obtain a character string, and extracts specific information from the read image based on a character string that matches a predetermined key character string among the obtained character strings (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-137094

[0004] However, in the case of the above-described image processing apparatus, when a correction object such as a strikethrough is entered superimposed on an already entered character string or the like and a correct character string or the like is entered again, there is a possibility that the entered information may not be appropriately extracted from the document image.

[0005] The present invention has been made in view of the above problems, and an object thereof is to obtain an image processing apparatus and an image processing method that appropriately extract entered information from a document image.

[0006] The image processing apparatus according to the present invention includes an entry object detection unit that detects an entry object entered in the document in a document image of the document, and (a) detects an entry object on which a correction object is superimposed among the detected entry objects, and (b) an entry information extraction unit that extracts entry information corresponding to the detected entry objects by excluding the entry objects on which the correction objects are superimposed.

[0007] The image processing method according to the present invention includes a step of detecting an entry object entered in the document in a document image of the document, and (a) detecting an entry object on which a correction object is superimposed among the detected entry objects, and (b) a step of extracting entry information corresponding to the detected entry objects by excluding the entry objects on which the correction objects are superimposed.

[0008] The image processing program according to the present invention causes a computer to function as the above-mentioned entry object detection unit and the above-mentioned entry information extraction unit.

[0009] According to the present invention, an image processing apparatus and an image processing method are obtained that can appropriately extract written information from a document image.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram showing an example of a document. Figure 3 is a diagram showing an example of a document image of a document in which entry objects have been entered into the document shown in Figure 2. Figure 4 is a diagram showing an example of form data 2b corresponding to the document shown in Figure 2. Figure 5 is a diagram showing an example of a selected object and a correction object among the entry objects detected in the document image shown in Figure 3. Figure 6 is a diagram showing an example of a handwritten string object among the entry objects detected in the document image shown in Figure 3. Figure 7 is a diagram showing an example of entry information (string) after correction based on a correction object has been made to the handwritten string object shown in Figure 6. Figure 8 is a diagram illustrating the extraction of entry information when there is no format in the form data 2b corresponding to the document image shown in Figure 3. Figure 9 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Embodiment 1.

[0014] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0015] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0016] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as form data 2b) necessary for the processing described later. The image processing program 2a is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2.

[0017] Form data 2b is data that indicates the position and size of the input area where the user should enter input objects (handwritten text, marks or stamps in selection options) within the document image of a specific document (form, application form, etc.).

[0018] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0019] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0020] In this case, the processing unit 1 operates according to the image processing program 2a as the processing units described above: the target image acquisition unit 11, the entry object detection unit 12, and the entry information extraction unit 13.

[0021] The target image acquisition unit 11 acquires document images as target images (raster image data) from the storage device 2, communication device 3, internal device 6, etc., and stores them in RAM or the like.

[0022] For example, this document image is a read image of a document in which an input object has been entered by the user. Figure 2 shows an example of a document. Figure 3 shows an example of a document image of the document shown in Figure 2 in which an input object has been entered.

[0023] Figure 4 shows an example of form data 2b corresponding to the document shown in Figure 2. Form data 2b is data that indicates the format for each document. Each format, as shown in Figure 4, for example, indicates the position and size of one or more printed strings pre-included in the document, the position and size of the input area for each item in one or more choices included in the document, the position and size of the input area for one or more handwritten strings included in the document, one or more items (keys), and the correspondence between each key and the choices, and between each key and the input area for the handwritten strings, as well as image data (reference image) of the document.

[0024] The entry object detection unit 12 detects entry objects in the document image described above. For example, the entry object detection unit 12 detects entry objects in the document image using an existing object detection method that uses a machine learning-trained learner, classifying them by their type (object type such as handwritten text, selection objects for selecting items in a choice (check marks, etc.), correction objects (strikethrough, etc.)), and identifies the position and size of the bounding box of each entry object. Alternatively, entry objects may be detected by pattern matching. Furthermore, strikethroughs as correction objects may be detected by pattern matching as a predetermined number of lines aligned with the direction of the text over which the strikethrough is superimposed.

[0025] The entry information extraction unit 13 (a) detects entry objects from among the detected entry objects that have correction objects superimposed on them, and (b) excludes the entry objects that have correction objects superimposed on them, and extracts entry information corresponding to the detected entry objects.

[0026] Figure 5 shows an example of selected and corrected objects among the input objects detected in the document image shown in Figure 3. For example, as shown in Figure 5, for selected objects that have a corrected object superimposed on them, the items selected in that selected object are excluded from the input objects from which input information should be extracted.

[0027] Figure 6 shows an example of a handwritten text object among the input objects detected in the document image shown in Figure 3. Figure 7 shows an example of input information (text) after corrections based on a correction object have been made to the handwritten text object shown in Figure 6.

[0028] For example, as shown in Figure 6, handwritten text objects are detected with the correction objects removed, and those with overlapping correction objects (in Figure 3, "3-27" and "Taro Tanaka") are excluded from the input objects from which input information should be extracted. Then, as shown in Figure 7, for example, the part of the text where the correction object overlaps is deleted, and the adjacent text is inserted in the position where the text was deleted.

[0029] Furthermore, if there is form data 2b (format) corresponding to the target document image, the entry information extraction unit 13 identifies the entry area where the entry object should be entered based on the form data 2b, for example, as shown in Figure 4, and extracts entry information corresponding to the detected entry object from the entry area in the document image. In this case, if there is no entry object superimposed on the entry area, the entry information extraction unit 13 searches for an entry object in a predetermined range around that entry area, identifies the found entry object as the entry object corresponding to that entry area, and extracts the entry information corresponding to that entry object as a value for the corresponding key.

[0030] If the input object is a handwritten string object, the input information extraction unit 13 performs character recognition processing to extract the string as input information.

[0031] Furthermore, if the input object is a selection object (such as a check mark or a circle mark) used to select an item from the options, the input information extraction unit 13 extracts strings adjacent to the selection object or strings over which the selection object overlaps as input information. Also, if the document includes options containing multiple items, and the input object includes a selection object indicating an item selected from those multiple items, the input information extraction unit 13 excludes the selection object over which the correction object overlaps and extracts the item selected from the multiple items by other selection objects as input information.

[0032] Furthermore, if the entry object is a seal impression image, the entry information extraction unit 13 excludes the seal impression image on which the correction object is superimposed and extracts the entry information from the seal impression image.

[0033] Figure 8 illustrates the extraction of information when there is no form data 2b corresponding to the document image shown in Figure 3. When there is no form data 2b (format) corresponding to the target document image, the information extraction unit 13 (a) detects all objects (printed strings and input objects), and (b) extracts items superimposed on selected objects, handwritten string objects, and correction objects, as shown in Figure 8, for example, and extracts the information along with the printed string, reflecting corrections made by the correction objects in the same manner as described above. The printed string is converted into text data by character recognition processing.

[0034] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 9 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0035] When the target image acquisition unit 11 acquires a document image (step S1), the entry object detection unit 12 detects entry objects in the document image, and the entry information extraction unit 13 determines whether or not there is a format in the form data 2b that has entry objects that match the position and size of the detected entry objects in a predetermined proportion or greater (step S2).

[0036] If there is a format in which the input objects match at a predetermined rate or higher, the input information extraction unit 13 extracts selected objects from the input objects detected in the document image and identifies the corresponding selection items for those selected objects, and also extracts correction objects from the input objects detected in the document image (step S3). Furthermore, the input information extraction unit 13 extracts handwritten string objects from the input objects detected in the document image (step S4), and performs character recognition processing on the handwritten string objects to extract the strings (step S5).

[0037] Then, the input information extraction unit 13 identifies the input information by reflecting the correction by the correction object as described above (step S6), and outputs the identified input information as the value of the corresponding item (key) (step S7). For example, this input information is stored in the storage device 2 or transmitted to another system.

[0038] On the other hand, if in step S2 it is determined that there is no format in which the input objects match at a predetermined rate or higher, the input information extraction unit 13 extracts all input objects detected in the document image (step S8), identifies the selection items corresponding to the selected objects among those input objects (step S9), and performs character recognition processing on the handwritten string objects among those input objects to extract the strings (step S10).

[0039] Then, the input information extraction unit 13 identifies the input information by reflecting the corrections made by the correction object among the input objects as described above (step S6), and outputs the identified input information as the value of the corresponding item (key) (step S7).

[0040] As described above, according to Embodiment 1, the entry object detection unit 12 detects entry objects written in the document image of the document. The entry information extraction unit 13 (a) detects entry objects from among the detected entry objects that have correction objects superimposed on them, and (b) excludes entry objects that have correction objects superimposed on them, and extracts entry information corresponding to the detected entry objects.

[0041] This ensures that corrections made on the document are reflected in the extraction of the entered information, allowing for the accurate extraction of the entered information from the document image. Furthermore, because corrections made on the document are reflected in the extraction of the entered information, there is no need to redo the same document if there are errors when filling it out.

[0042] Embodiment 2.

[0043] In Embodiment 2, the above-mentioned entry object includes a correction seal imprint image (an imprint image of a correction seal stamped adjacent to a correction object in a document), and the entry information extraction unit 13: (a) determines whether there is a correction seal imprint image within a predetermined range around the correction object; (b) if there is a correction seal imprint image within the predetermined range around the correction object, excludes the entry object overlapping the correction object, and extracts the entry information corresponding to the detected entry object.

[0044] If there is no correction seal imprint image within the predetermined range around the correction object, the entry information extraction unit 13 may extract the entry information corresponding to the detected entry object without excluding the entry object overlapping the correction object, or may stop extracting the entry information and notify the user of an error.

[0045] Also, in Embodiment 2, when there is a correction seal imprint image within the predetermined range around the correction object and the correction seal imprint image matches a registered correction seal imprint image registered in advance, the entry information extraction unit 13 excludes the entry object overlapping the correction object, extracts the entry information corresponding to the detected entry object, and (c) when the registered correction seal imprint image is not registered, determines whether to register the correction seal imprint image as the registered correction seal imprint image based on a user operation.

[0046] Here, for example, the image data of the registered correction seal imprint image is stored in the storage device 2 in advance, and the entry information extraction unit 13 reads out the registered correction seal imprint image (image data) from the storage device 2 and determines whether the correction seal imprint image matches the registered correction seal imprint image.

[0047] Also, when the registered correction seal impression image is not registered (that is, when the image data of the registered correction seal impression image is not stored in the storage device 2), the entry information extraction unit 13 displays, on the display device 4, a dialog for asking the user about the correction seal impression image and whether to register this correction seal impression image as the registered correction seal impression image. Then, when detecting a user operation on the input device 5 (a user operation indicating whether to register the correction seal impression image as the registered correction seal impression image), the entry information extraction unit 13 determines whether to register the correction seal impression image as the registered correction seal impression image based on the user operation. If it is determined to register the correction seal impression image as the registered correction seal impression image, the image data of the correction seal impression image is stored in the storage device 2.

[0048] As described above, according to the second embodiment, when a correction object is detected, extraction of entry information corresponding to whether a correction seal has been stamped is executed, so that the information entered in the document image is appropriately extracted.

[0049] Note that since other configurations and operations of the image processing apparatus according to the second embodiment are the same as those of the first embodiment, the description thereof is omitted.

[0050] Various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, it is intended that such changes and modifications be included in the claims.

[0051] For example, in the above embodiments 1 and 2, if there are no writing objects superimposed on the writing area, the writing information extraction unit 13 may: (a) if a correction object is detected in the writing area, search for writing objects in a predetermined range around the writing area, set the found writing object as the writing object corresponding to the writing area, and extract writing information corresponding to the writing object; and (b) if no correction object is detected in the writing area, not search for writing objects in a predetermined range around the writing area, and determine that no writing objects were found for the writing area.

[0052] The present invention is applicable, for example, to an image processing apparatus.

Claims

1. An image processing apparatus comprising: an entry object detection unit that detects entry objects written in a document image of a document; and an entry information extraction unit that (a) detects entry objects from among the detected entry objects that have correction objects superimposed on them, and (b) excludes the entry objects that have correction objects superimposed on them and extracts entry information corresponding to the detected entry objects.

2. The image processing apparatus according to claim 1, wherein the entry information extraction unit (a) identifies an entry area where the entry object should be entered based on form data, and extracts entry information corresponding to the detected entry object from the entry area in the document image, and (b) if there is no entry object superimposed on the entry area, searches for an entry object in a predetermined range around the entry area, sets the found entry object as the entry object corresponding to the entry area, and extracts entry information corresponding to the entry object.

3. The image processing apparatus according to claim 1, characterized in that the document includes a selection of multiple items, the entry object includes a selection object indicating a string selected from the multiple items, and the entry information extraction unit excludes the selection object on which the correction object is superimposed and extracts the item selected from the multiple items by the selection object as the entry information.

4. The image processing apparatus according to claim 1, wherein the entry object includes a seal impression image, and the entry information extraction unit extracts the entry information from the seal impression image, excluding the seal impression image on which the correction object is superimposed.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the entry object includes a correction seal impression image, and the entry information extraction unit (a) determines whether or not the correction seal impression image is present in a predetermined range around the correction object, and (b) if the correction seal impression image is present in a predetermined range around the correction object, it excludes the entry object on which the correction object is superimposed and extracts entry information corresponding to the detected entry object.

6. The image processing apparatus according to claim 5, characterized in that the entry information extraction unit (a) determines whether or not the correction seal impression image exists in a predetermined range around the correction object; (b) if the correction seal impression image exists in a predetermined range around the correction object and the correction seal impression image matches a registered correction seal impression image that has been registered in advance, it excludes the entry object on which the correction object is superimposed and extracts entry information corresponding to the detected entry object; and (c) if the registered correction seal impression image has not been registered, it determines whether or not to register the correction seal impression image as the registered correction seal impression image based on user operation.

7. An image processing method characterized by comprising the steps of: (a) detecting an entry object written in a document in a document image of a document; (b) detecting an entry object in which a correction object is superimposed among the detected entry objects; and (c) excluding the entry object in which the correction object is superimposed and extracting entry information corresponding to the detected entry object.

Images