Image processing device and image processing method

Abstract

A form region identification unit (12) identifies a form region in a target image on the basis of form data. A handwritten character string region detection unit (13) detects a handwritten character string region in the form region. A handwritten character string extraction unit (14) subjects an image of the handwritten character string region to handwritten character recognition processing, and extracts a handwritten character string corresponding to the form region. When there is a character string portion for which handwritten character string extraction has failed in handwritten character recognition processing, the handwritten character string extraction unit (14) enlarges an image of the character string portion for which handwritten character string extraction has failed, subjects the enlarged image to handwritten character recognition processing, and extracts the handwritten character string in said character string portion.

Description

Image processing apparatus and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] A certain image forming apparatus detects (a) a character string area of handwritten characters included in a read image obtained from a document in which a handwritten character string is entered in a form area by a user, (b) specifies position information of the form area where the handwritten character string should be described based on form data, (c) specifies a character string area within the form area in the read image, and (d) extracts a character string from the character string area (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-29228

[0004] When a handwritten character string is entered in the form area, it may be entered in small characters. In that case, in the above-described image forming apparatus, there is a possibility that small characters may not be correctly extracted by handwritten character recognition. For example, when entering an address in an address entry field (form area), when entering a building name or room number as well, the number of characters increases, so it may be entered in small characters.

[0005] The present invention has been made in view of the above problems, and an object thereof is to obtain an image processing apparatus and an image processing method that can correctly extract a handwritten character string even if there are small characters in the handwritten character string entered in the form area.

[0006] The image processing apparatus according to the present invention includes a form area specifying unit that specifies a form area in a target image based on form data, a handwritten character string area detecting unit that detects a handwritten character string area in the form area, and a handwritten character string extracting unit that executes a handwritten character recognition process on an image of the handwritten character string area and extracts a handwritten character string corresponding to the form area. Then, when there is a character string portion for which the extraction of the handwritten character string fails in the handwritten character recognition process, the handwritten character string extracting unit (a) enlarges an image of the character string portion for which the extraction of the handwritten character string fails, and (b) executes the handwritten character recognition process on the enlarged image to extract a handwritten character string of the character string portion.

[0007] The image processing method according to the present invention comprises the steps of: identifying a form region in a target image based on form data; detecting a handwritten character region in the form region; and extracting a handwritten character by performing handwritten character recognition processing on the image of the handwritten character region to extract the handwritten character corresponding to the form region. In the handwritten character by handwriting

[0008] The image processing program according to the present invention causes a computer to function as the form area identification unit, the handwritten character area detection unit, and the handwritten character extraction unit described above.

[0009] According to the present invention, an image processing device and an image processing method are obtained that can correctly extract handwritten characters even if they contain small characters in the form area.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram illustrating the extraction of a string from a handwritten string area in a form area. Figure 3 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0014] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as form data 2b) necessary for the processing described later. The image processing program is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2. The form data 2b is data that indicates the position and size of a form area within a specific target image where a handwritten string should be entered by the user.

[0016] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0017] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0018] In this configuration, the arithmetic processing unit 1 operates according to the image processing program 2a as the aforementioned processing units: the target image acquisition unit 11, the form area identification unit 12, the handwritten character area detection unit 13, and the handwritten character extraction unit 14.

[0019] The target image acquisition unit 11 acquires the target image (raster image data) from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM or the like.

[0020] For example, this image is a scanned image of a given document that includes a form area where a user has entered handwritten text. In this case, the form area is a rectangular area enclosed by a grid.

[0021] The form area identification unit 12 identifies the form area (specifically, its position and size) in the target image based on the form data 2b.

[0022] The handwritten string area detection unit 13 detects handwritten string areas in the form area described above. For example, the handwritten string area detection unit 13 performs character recognition processing using a machine learning-trained learner and detects handwritten string areas in the target image using an existing object detection method. Figure 2 is a diagram illustrating the extraction of strings from handwritten string areas in the form area. As shown in Figure 2, the form area is, for example, a field for entering an address.

[0023] The handwritten string extraction unit 14 performs handwritten character recognition processing on the image of the handwritten string area to extract the handwritten string corresponding to the form area. Here, the handwritten string extraction unit 14 performs handwritten character recognition processing using a machine learning model that has been used for existing object detection, and outputs the characters (character codes, i.e., text data) contained in the handwritten string area and the confidence level of those characters from the image of the handwritten string area.

[0024] Furthermore, if there is a portion of the string that the handwritten character recognition process failed to extract, the handwritten character extraction unit 14 (a) enlarges the image of the portion of the string that failed to extract the handwritten character, and (b) performs the handwritten character recognition process on the enlarged image to extract the handwritten character from that portion of the string.

[0025] Here, the handwritten string extraction unit 14 considers the portion of the string whose confidence level is below a predetermined threshold as the portion of the string for which handwritten string extraction failed.

[0026] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 3 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0027] When the target image acquisition unit 11 acquires a document image (step S1), the form area identification unit 12 identifies the form area (step S2), and the handwritten character area detection unit 13 detects the handwritten character area within the form area (step S3).

[0028] Then, the handwritten character extraction unit 14 performs handwritten character recognition processing on the image of the handwritten character region (step S4).

[0029] The handwritten character extraction unit 14 determines whether or not there are parts of the handwritten character recognition process that have low confidence (parts where the confidence level is below a predetermined threshold) (step S5).

[0030] If there are parts of the string with low reliability, the handwritten string extraction unit 14 enlarges the image of the part of the string (step S6) and performs handwritten character recognition processing on the enlarged image (step S7).

[0031] Here, the handwritten character extraction unit 14 determines whether the confidence level of the character obtained by the handwritten character recognition process on the enlarged image is above a predetermined threshold (step S8).

[0032] If the confidence level is above a predetermined threshold, the handwritten character extraction unit 14 generates a string by concatenating the highly confident string portion obtained in the initial (step S4) handwritten character recognition process with the string portion obtained from the enlarged image, as shown in Figure 2, and outputs the string (text data) as the string corresponding to the form area (step S9). For example, the string (text data) may be saved or transmitted to another system.

[0033] On the other hand, in step S8, if the confidence level is below a predetermined threshold, the handwritten string extraction unit 14 performs error processing (discarding the part of the string with low confidence level, notifying the user, etc.) (step S10).

[0034] Furthermore, if it is determined in step S5 that there are no low-confidence character parts, the handwritten character extraction unit 14 outputs the character string (text data) obtained in the handwritten character recognition process in step S4 as the character string corresponding to the form area (step S9).

[0035] As described above, according to the above embodiment, the form area identification unit 12 identifies the form area in the target image based on the form data. The handwritten character area detection unit 13 detects the handwritten character area in the form area. The handwritten character extraction unit 14 performs handwritten character recognition processing on the image of the handwritten character area to extract the handwritten character corresponding to the form area. If there is a part of the character that the handwritten character recognition processing failed to extract, the handwritten character extraction unit 14 (a) enlarges the image of the part of the character that failed to extract the handwritten character, and (b) performs handwritten character recognition processing on the enlarged image to extract the handwritten character of that part of the character.

[0036] This ensures that even if there are small characters in the handwritten text entered in the form area, it will be extracted correctly.

[0037] Furthermore, various changes and modifications to the embodiments described above will be obvious to those skilled in the art. Such changes and modifications may be made without deviating from the spirit and scope of the subject matter and without diminishing the intended advantages. In other words, such changes and modifications are intended to be included in the claims.

[0038] The present invention is applicable, for example, to an image processing apparatus.

Claims

1. An image processing device comprising: a form area identification unit that identifies a form area in a target image based on form data; a handwritten string area detection unit that detects a handwritten string area in the form area; and a handwritten string extraction unit that performs handwritten character recognition processing on the image of the handwritten string area to extract the handwritten string corresponding to the form area, wherein, if there is a string portion for which the handwritten string was not extracted in the handwritten character recognition processing, (a) enlarges the image of the string portion for which the handwritten string was not extracted, and (b) performs the handwritten character recognition processing on the enlarged image to extract the handwritten string of that string portion.

2. The image processing apparatus according to claim 1, characterized in that the handwritten character extraction unit performs the handwritten character recognition process using a machine learning-trained learner, the learner outputs the characters and confidence levels included in the image of the handwritten character region, and the handwritten character extraction unit designates the portion of the character whose confidence level is below a predetermined threshold as the portion of the character for which the handwritten character extraction failed.

3. The image processing apparatus according to claim 1, characterized in that the form area is a field for entering an address.

4. An image processing method comprising: a step of identifying a form area in a target image based on form data; a step of detecting a handwritten character area in the form area; and a handwritten character extraction step of performing handwritten character recognition processing on the image of the handwritten character area to extract the handwritten character corresponding to the form area, wherein in the handwritten character extraction step, if there is a part of the character that failed to be extracted in the handwritten character recognition processing, (a) an enlargement of the image of the part of the character that failed to be extracted, and (b) the handwritten character recognition processing performed on the enlarged image to extract the handwritten character of that part of the character.

Images