Image processing device and image processing method

Abstract

A form area specification unit (12) identifies the position and size of a form area in a target image on the basis of form data 2b. A handwritten character string area detection unit (13) detects a handwritten character string area in the form area. A handwritten character string extraction unit (14) extracts a handwritten character string in the handwritten character string area corresponding to the form area. In particular, when an end of the handwritten character string area matches an end of the form area in a prescribed direction, the handwritten character string extraction unit (14) expands the handwritten character string area along the prescribed direction by a specific expansion width, and extracts a handwritten character string corresponding to the form area from the expanded handwritten character string area.

Description

Image processing apparatus and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] A certain image forming apparatus detects a character string area of handwritten characters included in a read image obtained from a document in which a handwritten character string is entered in a form area by a user, (b) specifies position information of the form area where the handwritten character string should be described based on form data, (c) specifies the character string area within the form area in the read image, and (d) extracts a character string from the character string area (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2022-29,228

[0004] However, in the above-described image forming apparatus, when a user enters a handwritten character string so as to protrude from the form area of the document, the portion outside the form area of the handwritten character string is not extracted from the read image, and there is a possibility that the character string entered by the user may not be appropriately extracted.

[0005] The present invention has been made in view of the above problems, and an object thereof is to obtain an image processing apparatus and an image processing method in which a handwritten character string entered by a user is appropriately extracted from a target image.

[0006] The image processing apparatus according to the present invention includes a form area specifying unit that specifies the position and size of a form area in a target image based on form data, a handwritten character string area detecting unit that detects a handwritten character string area in the form area, and a handwritten character string extracting unit that extracts a handwritten character string for the handwritten character string area corresponding to the form area. Then, when the end of the handwritten character string area in a predetermined direction coincides with the end of the form area, the handwritten character string extracting unit (a) expands the handwritten character string area along the predetermined direction by a specific expansion width, and (b) extracts a handwritten character string corresponding to the form area from the expanded handwritten character string area.

[0007] The image processing method according to the present invention comprises the steps of: identifying the position and size of a form area in a target image based on form data; detecting a handwritten character area in the form area; and extracting a handwritten character for the handwritten character area corresponding to the form area. In the handwritten character extraction step, if the edge of the handwritten character area in a predetermined direction coincides with the edge of the form area, (a) the handwritten character area is expanded along the predetermined direction by a specific expansion width, and (b) the handwritten character corresponding to the form area is extracted from the expanded handwritten character area.

[0008] The image processing program according to the present invention causes a computer to function as the form area identification unit, the handwritten character area detection unit, and the handwritten character extraction unit described above.

[0009] According to the present invention, an image processing apparatus and an image processing method are obtained that appropriately extract handwritten strings entered by a user from a target image.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram showing an example of the expansion of a handwritten character area. Figure 3 is a diagram showing another example of the expansion of a handwritten character area. Figure 4 is a diagram showing an example of the expansion of a handwritten character area when the handwritten character area includes a character cancel. Figure 5 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0014] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as form data 2b) necessary for the processing described later. The image processing program is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2. The form data 2b is data that indicates the position and size of a form area within a specific target image where a handwritten string should be entered by the user.

[0016] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0017] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0018] In this case, the arithmetic processing unit 1 operates as the aforementioned processing units: the target image acquisition unit 11, the form area identification unit 12, the handwritten character area detection unit 13, and the handwritten character extraction unit 14.

[0019] The target image acquisition unit 11 acquires the target image (raster image data) from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM or the like.

[0020] For example, this image is a scanned image of a given document that includes a form area where a user has entered handwritten text. In this case, the form area is a rectangular area enclosed by a grid.

[0021] The form area identification unit 12 identifies the position and size of the form area in the target image based on the form data 2b.

[0022] The handwritten character area detection unit 13 detects handwritten character areas within the form area described above. For example, the handwritten character area detection unit 13 uses a machine learning-trained learner to perform character recognition processing to detect handwritten character areas in the target image using existing methods, and identifies the handwritten character areas within the form area from among the detected handwritten character areas. Alternatively, the handwritten character area detection unit 13 may perform the above-mentioned character recognition processing only on the portion of the target image within the form area to detect handwritten character areas within the form area.

[0023] The handwritten string extraction unit 14 extracts handwritten strings for the handwritten string area corresponding to the form area. For example, the handwritten string extraction unit 14 obtains the text data of the handwritten string from the result of the character recognition process described above and stores it in association with the form area (for example, the identifier of the form area).

[0024] Figure 2 shows an example of the expansion of the handwritten text area. Furthermore, as shown in Figure 2, for example, if the edge of the handwritten text area and the edge of the form area coincide in a predetermined direction (main scanning direction (right or left) or sub-scanning direction (up or down)), the handwritten text extraction unit 14 (a) expands the handwritten text area by a specific expansion width along that predetermined direction, and (b) extracts the handwritten text corresponding to the form area from the expanded handwritten text area. In other words, the handwritten text contained in the expanded handwritten text area is extracted.

[0025] Figure 3 shows another example of the expansion of the handwritten text area. In particular, in this embodiment, when the end of the handwritten text area and the end of the form area coincide in a predetermined direction (for example, in Figure 2) and when the distance between the end of the handwritten text area and the end of the form area in a predetermined direction is less than a predetermined value (for example, in Figure 3), the handwritten text extraction unit 14 (a) expands the handwritten text area along its predetermined direction by a specific expansion width, and (b) extracts the handwritten text corresponding to the form area from the expanded handwritten text area.

[0026] The predetermined value mentioned above may be set to correspond to the character size of the handwritten string. For example, the predetermined value mentioned above is set to be the same as the width of one character in the handwritten string in the predetermined direction mentioned above. In other words, for example, the width of one character in the predetermined direction mentioned above is set to the character size.

[0027] Furthermore, the above-mentioned expansion width may be a fixed length or may be set by the user. Figure 4 shows an example of expansion of the handwritten string area when the handwritten string area includes a character strikethrough. For example, as shown in Figure 4, even when the handwritten character area includes a character strikethrough (i.e., a part where a strikethrough line or the like is superimposed on the character), the handwritten string extraction unit 14 may (a) expand the handwritten string area by a specific expansion width along its predetermined direction, and (b) extract the handwritten string corresponding to the form area from the expanded handwritten string area. For example, in this case, the expansion width is set according to the number of characters in the character strikethrough. In Figure 4, since the number of characters in the character strikethrough is 1, the expansion width is set to the size of one character (character size × number of characters). Alternatively, the expansion width may be increased by the product of the character size and the number of characters in the character strikethrough. Furthermore, the character strikethrough can be detected together with the above-mentioned learning unit, and the number of characters can be derived, for example, from the size (width) of the character strikethrough and the character size (character width).

[0028] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 5 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0029] When the target image acquisition unit 11 acquires a target image (step S1), the form area identification unit 12 identifies the position and size of the form area in the target image based on the form data 2b corresponding to the target image (step S2), and the handwritten character area detection unit 13 detects the handwritten character area in the form area (step S3).

[0030] The handwritten string extraction unit 14 determines whether or not to expand the handwritten string area corresponding to the form area, as described above, based on the positional relationship of the edges between the handwritten string area and the form area, character cancellation, etc. (step S4).

[0031] If the handwritten string extraction unit 14 determines that it does not need to expand the handwritten string area corresponding to the form area, it extracts the handwritten string from the current handwritten string area (step S5).

[0032] On the other hand, if the handwritten string extraction unit 14 determines that it is necessary to expand the handwritten string area corresponding to the form area, it first sets the expansion width as described above (step S6), and then expands the handwritten string area by that expansion width (step S7). After that, the handwritten string extraction unit 14 extracts the handwritten string from the expanded handwritten string area (step S5).

[0033] As described above, according to the above embodiment, the form area identification unit 12 identifies the position and size of the form area in the target image based on the form data 2b. The handwritten character area detection unit 13 detects the handwritten character area in the form area. The handwritten character extraction unit 14 extracts the handwritten character for the handwritten character area corresponding to the form area. In particular, if the edge of the handwritten character area in a predetermined direction coincides with the edge of the form area, the handwritten character extraction unit 14 (a) expands the handwritten character area along its predetermined direction by a specific expansion width, and (b) extracts the handwritten character corresponding to the form area from the expanded handwritten character area.

[0034] This ensures that even if the user's handwritten text extends beyond the form area, the handwritten text entered by the user is properly extracted from the target image.

[0035] Furthermore, various changes and modifications to the embodiments described above will be obvious to those skilled in the art. Such changes and modifications may be made without deviating from the spirit and scope of the subject matter and without diminishing the intended advantages. In other words, such changes and modifications are intended to be included in the claims.

[0036] For example, in the above embodiment, if the handwritten text in the form area where the address should be entered ends between "ban" and "go", the handwritten text area may be expanded as described above.

[0037] Furthermore, in the above embodiment, the image processing device is a device used directly by the user, but it may also be a device such as a server (e.g., a cloud server) used indirectly by the user via a network. In that case, the user interface of the client device is used instead of the display device 4 and the input device 5.

[0038] The present invention is applicable, for example, to an image processing apparatus.

Claims

1. An image processing apparatus comprising: a form area identification unit that identifies the position and size of a form area in a target image based on form data; a handwritten string area detection unit that detects a handwritten string area in the form area; and a handwritten string extraction unit that extracts a handwritten string from the handwritten string area corresponding to the form area, wherein the handwritten string extraction unit, when the edge of the handwritten string area in a predetermined direction coincides with the edge of the form area, (a) expands the handwritten string area along the predetermined direction by a specific expansion width, and (b) extracts a handwritten string corresponding to the form area from the expanded handwritten string area.

2. The image processing apparatus according to claim 1, wherein the handwritten string extraction unit, when the end of the handwritten string area in a predetermined direction coincides with the end of the form area, and when the distance between the end of the handwritten string area in the predetermined direction and the end of the form area is less than a predetermined value, (a) expands the handwritten string area along the predetermined direction by a specific expansion width, and (b) extracts the handwritten string corresponding to the form area from the expanded handwritten string area.

3. The image processing apparatus according to claim 2, characterized in that the predetermined value is set in accordance with the character size of the handwritten string.

4. The image processing apparatus according to claim 1, characterized in that, if the handwritten character area includes a character cancellation portion, (a) the handwritten character area is expanded by the expansion width along the predetermined direction, and (b) the handwritten character corresponding to the form area is extracted from the expanded handwritten character area, wherein the expansion width is set in accordance with the number of characters in the character cancellation portion.

5. An image processing method comprising: a step of identifying the position and size of a form area in a target image based on form data; a step of detecting a handwritten character area in the form area; and a handwritten character extraction step of extracting a handwritten character for the handwritten character area corresponding to the form area, wherein in the handwritten character extraction step, if the edge of the handwritten character area in a predetermined direction coincides with the edge of the form area, (a) the handwritten character area is expanded along the predetermined direction by a specific expansion width, and (b) the handwritten character corresponding to the form area is extracted from the expanded handwritten character area.

Images