Image processing device and image processing method

Abstract

A character region detection unit (12) detects a handwritten character region in a document image. A character extraction unit (13) identifies, from among the detected handwritten character regions, a handwritten character region corresponding to the handwritten character region designated by form data (2b), and extracts characters from the identified handwritten character region. A form data update unit (14) updates the form data (2b) by enlarging the handwritten character region designated by the form data so as to include the detected handwritten character region.

Description

Image processing apparatus and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] A certain image processing apparatus identifies, among string regions specified by form data corresponding to a form having a region where a string is handwritten, a string region whose position is similar to the string region detected in the read image of the form, and extracts the string of a predetermined item associated with the identified string region from the string region detected in the read image of the form (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2022-29228

[0004] However, in the case of the above-described image processing apparatus, for a string region in which handwritten characters are entered, there are variations in the entry position depending on the person entering the characters. Therefore, in a document image such as a form, the position of the string region specified by the form data and the position of the string region detected in the read image are separated, and there is a possibility that the handwritten characters cannot be appropriately extracted from the handwritten character entry region.

[0005] The present invention has been made in view of the above problems, and an object thereof is to obtain an image processing apparatus, an image processing method, and an image processing program that can appropriately extract handwritten characters from a handwritten string region specified by form data.

[0006] The image processing apparatus according to the present invention includes a character region detection unit that detects a handwritten character region in a document image, a character extraction unit that identifies a handwritten character region corresponding to the handwritten character region specified by form data among the detected handwritten character regions and extracts characters from the identified handwritten character region, and a form data update unit that updates the form data by expanding the handwritten character region specified by the form data so as to include the detected handwritten character region.

[0007] The image processing method according to the present invention comprises the steps of: detecting handwritten character regions in a document image; identifying a handwritten character region from among the detected handwritten character regions that corresponds to a handwritten character region specified by form data; extracting characters from the identified handwritten character region; and updating the form data by expanding the handwritten character region specified by the form data to include the detected handwritten character region.

[0008] The image processing program according to the present invention causes a computer to function as the character area detection unit, the character extraction unit, and the form data update unit described above.

[0009] According to the present invention, an image processing device and an image processing method are obtained that appropriately extract handwritten characters from a handwritten character string area specified by form data.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram showing an example of a document. Figure 3 is a diagram showing an example of a document image of a document in which handwritten characters have been added to the document shown in Figure 2. Figure 4 is a diagram illustrating the position and size of the character area detected in the document image shown in Figure 3. Figure 5 is a diagram illustrating an example of the printed character area and handwritten character area specified in each format in the form data 2b. Figure 6 is a diagram showing another example of a document image of a document in which handwritten characters have been added to the document shown in Figure 2. Figure 7 is a diagram illustrating the position and size of the character area detected in the document image shown in Figure 6. Figure 8 is a diagram showing a handwritten character area detected in the document image that is shifted from the handwritten character area specified by the form data 2b, among the handwritten character areas shown in Figure 7. Figure 9 is a diagram showing an example of an enlarged handwritten character area in the form data 2b. Figure 10 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1. Figure 11 is a diagram illustrating the frame surrounding the handwritten character area detected in the document image. Figure 12 is a diagram illustrating an example of enlargement of the handwritten character area in the form data 2b in Embodiment 2.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Embodiment 1.

[0014] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0015] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0016] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as form data 2b) necessary for the processing described later. The image processing program is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2. The form data 2b is data that indicates the position and size of character areas within the document image of a specific document (such as a form or application form) where handwritten text should be entered by the user.

[0017] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0018] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0019] In this configuration, the processing unit 1 operates according to the image processing program 2a as the processing units described above: the target image acquisition unit 11, the character area detection unit 12, the character extraction unit 13, and the form data update unit 14.

[0020] The target image acquisition unit 11 acquires document images as target images (raster image data) from the storage device 2, communication device 3, internal device 6, etc., and stores them in RAM or the like.

[0021] For example, this document image is a read image of a given document that contains a text area where a user has written handwritten text. Here, the text area is a rectangular area enclosed by a border (ruled line). Figure 2 shows an example of a document. Figure 3 shows an example of a document image of the document shown in Figure 2 with handwritten text added.

[0022] The character region detection unit 12 detects handwritten character regions in a document image. For example, the character region detection unit 12 uses an existing object detection method with a machine learning-trained learner to detect handwritten character objects in a document image, and identifies the position and size of the bounding box of those objects as the character region of the handwritten character (i.e., the handwritten character region).

[0023] Figure 4 is a diagram illustrating the position and size of the character regions detected in the document image shown in Figure 3. For example, as shown in Figure 4, the character region detection unit 12 distinguishes and identifies printed character regions and handwritten character regions from each other using an existing object detection method that uses a machine learning-trained learner.

[0024] The character extraction unit 13 identifies the handwritten character area that corresponds to the handwritten character area specified by the form data 2b from among the detected handwritten character areas, and extracts characters from the identified handwritten character area that are associated with a predetermined item specified by the form data 2b.

[0025] Figure 5 illustrates an example of the printable character area and handwritten character area specified in each format of form data 2b.

[0026] Specifically, form data 2b is data that indicates the format for each document. Each format, as shown in Figure 5 for example, indicates the position and size of one or more printed text areas included in the document, the position and size of one or more handwritten text areas included in the document, one or more items, and the correspondence between each item and the handwritten text area, as well as image data (reference image) of the document.

[0027] Therefore, the character extraction unit 13 reads the document format corresponding to the acquired document image in the form data 2b, selects the corresponding handwritten character area from the handwritten character areas detected in the document image for each handwritten character area specified by that format, performs character recognition processing on that handwritten character area, and extracts the character as the value of the item corresponding to that handwritten character area.

[0028] In Embodiment 1, for example, the character extraction unit 13 identifies handwritten character regions from among the detected handwritten character regions in which the proportion of the overlap with the handwritten character region specified by the form data 2b exceeds a predetermined value, and extracts characters from the identified handwritten character regions. Alternatively, the character extraction unit 13 may identify handwritten character regions from among the detected handwritten character regions within a predetermined distance range from the position of the handwritten character region specified by the form data 2b, and extract characters from the identified handwritten character regions.

[0029] Specifically, the character extraction unit 13 performs handwritten character recognition processing on the image of the handwritten character region to extract the string. Here, the character extraction unit 13 performs handwritten character recognition processing using a machine learning-trained learner and outputs the characters (character codes, i.e., text data) contained in the handwritten string region from the image of the handwritten character region.

[0030] Figure 6 shows another example of a document image in which handwritten text has been added to the document shown in Figure 2. Figure 7 is a diagram illustrating the position and size of the character area detected in the document image shown in Figure 6. Figure 8 shows a handwritten character area detected in the document image that is shifted from the handwritten character area specified by form data 2b, among the handwritten character areas shown in Figure 7. Figure 9 shows an example of an enlarged handwritten character area in form data 2b.

[0031] The form data update unit 14 updates the form data 2b by expanding the handwritten character area specified by the form data 2b to include the detected handwritten character area, for example, as shown in Figures 6 to 8, if the handwritten character area specified by the form data 2b does not include the detected handwritten character area, as shown in Figure 9. In this case, the handwritten character area in the form data 2b is changed to a bounding rectangle encompassing the handwritten character area before expansion and the detected handwritten character area.

[0032] Furthermore, the form data update unit 14 does not update the form data 2b if the handwritten character area specified by the form data 2b, for example, as shown in the document images in Figures 3 and 4, encompasses the detected handwritten character area.

[0033] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 10 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0034] Once the target image acquisition unit 11 acquires a document image (step S1), the character area detection unit 12 detects printed character areas and handwritten character areas in the document image (step S2).

[0035] The character extraction unit 13 determines whether or not there is a format in the form data 2b that has printed character areas and handwritten character areas that match the position and size of the detected printed character areas and handwritten character areas by a predetermined percentage or more (step S3).

[0036] If there is a format in which the printed character area and the handwritten character area match at a predetermined rate or higher, the character extraction unit 13 identifies the detected handwritten character area corresponding to each handwritten character area in that format and determines whether or not there are pairs of handwritten character areas whose positions are offset from each other (step S4).

[0037] If there is a misalignment between at least one handwritten character area in the format and the detected handwritten character area, the character extraction unit 13 enlarges the misaligned handwritten character area as described above and updates the form data 2b (step S5).

[0038] On the other hand, if there are no handwritten character areas with such misalignment, the character extraction unit 13 does not update the form data 2b.

[0039] The character extraction unit 13 then performs character recognition processing on the image of the detected handwritten character area to extract the characters (text data) (step S6), and outputs them as the value of the item associated with that handwritten area in the form data 2b (step S7). For example, the characters (text data) may be saved or sent to another system.

[0040] On the other hand, in step S3, if there is no format in which the printed character area and the handwritten character area match in a predetermined proportion or more, the form data update unit 14 adds a new format including the printed character area and the handwritten character area detected in the document image to the form data 2b (step S8). At this time, for example, images of the printed character area and the handwritten character area (or characters obtained by character recognition processing) are displayed on the display device 4, the characters obtained from the image of the printed character area are set as items, the handwritten character area corresponding to each item is manually specified by the user, and that handwritten character area is associated with that item and included in the new format.

[0041] After that, the character extraction unit 13 performs character recognition processing on the image of the detected handwritten character area to extract characters (text data) (step S6), and outputs them as the values of the items associated with the handwritten area in the form data 2b (step S7).

[0042] As described above, according to the first embodiment, the character area detection unit 12 detects a handwritten character area in the document image. The character extraction unit 13 identifies the handwritten character area corresponding to the handwritten character area specified by the form data 2b among the detected handwritten character areas, and extracts characters from the identified handwritten character area. The form data update unit 14 updates the form data 2b by expanding the handwritten character area specified by the form data 2b so as to include the detected handwritten character area.

[0043] As a result, the handwritten character area in the form data 2b is adaptively expanded, so that the handwritten characters are appropriately extracted from the string area where the characters are written by hand.

[0044] Embodiment 2.

[0045] FIG. 11 is a diagram for explaining a frame line surrounding a handwritten character area detected in a document image. In the second embodiment, for example, as shown in FIG. 11, when there is a frame line surrounding the detected handwritten character area, the character extraction unit 13 identifies the detected handwritten character area as the handwritten character area corresponding to the handwritten character area located inside the frame line among the handwritten character areas specified by the form data 2b. In FIG. 11, when there is a frame line surrounding a handwritten character area such as "Yamamoto Kazuo", among the handwritten character areas in the form data 2b, those inside the frame line are associated with the detected handwritten character area. Such a frame line is detected, for example, by pattern matching or a learned machine learning device.

[0046] FIG. 12 is a diagram for explaining an example of enlargement of a handwritten character area in the form data 2b in the second embodiment. Further, in the second embodiment, when there is a frame line surrounding the detected handwritten character area, the form data update unit 14 enlarges the handwritten character area of the form data corresponding to the detected handwritten character area to the area surrounded by the frame line, for example, as shown in FIG. 12, to update the form data 2b.

[0047] Note that since the other configurations and operations of the image processing apparatus according to the second embodiment are the same as those of the first embodiment, the description thereof is omitted.

[0048] Various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, such changes and modifications are intended to be included within the scope of the claims.

[0049] For example, in the above-described first and second embodiments, character extraction may not be performed for handwritten characters in a document image on which a correction object such as a strikethrough is superimposed. For example, such a correction object is detected by a learned machine learning device in the same manner as handwritten characters.

[0050] Further, in the second embodiment, when the handwritten character area detected in the document image protrudes outside the above-described frame line, the corresponding handwritten character area in the form data 2b may be enlarged to include the detected handwritten character area up to the outside of the frame line. However, when enlarging the handwritten character area in the form data 2b, the handwritten character area is enlarged with an enlargement width that does not overlap with other handwritten character areas.

[0051] The present invention is applicable to, for example, an image processing apparatus.

Claims

1. An image processing apparatus comprising: a character area detection unit that detects handwritten character areas in a document image; a character extraction unit that identifies a handwritten character area from the detected handwritten character areas that corresponds to a handwritten character area specified by form data, and extracts characters from the identified handwritten character area; and a form data update unit that updates the form data by expanding the handwritten character area specified by the form data to include the detected handwritten character area.

2. The image processing apparatus according to claim 1, characterized in that the character extraction unit identifies a handwritten character area within a predetermined distance from the position of the handwritten character area specified by the form data among the detected handwritten character areas, and extracts characters from the identified handwritten character area.

3. The image processing apparatus according to claim 1, characterized in that, if the character extraction unit has a frame surrounding the detected handwritten character area, it identifies the detected handwritten character area as a handwritten character area located inside the frame among the handwritten character areas specified by the form data.

4. The image processing apparatus according to claim 3, wherein the form data update unit updates the form data by expanding the handwritten character area of ​​the form data corresponding to the detected handwritten character area into the area enclosed by the frame, if there is a frame surrounding the detected handwritten character area.

5. An image processing method characterized by comprising: detecting handwritten character regions in a document image; identifying a handwritten character region from among the detected handwritten character regions that corresponds to a handwritten character region specified by form data, and extracting characters from the identified handwritten character region; and updating the form data by expanding the handwritten character region specified by the form data to include the detected handwritten character region.

Images