Image processing device and image processing method

Abstract

A handwritten character string extraction unit (14), when having detected a correction object within a handwritten character string region, determines whether another handwritten character string region is present within a prescribed replacement character string detection range adjacent to the correction object, and, if the other handwritten character string region is present within the replacement character string detection range, sets as a handwritten character string corresponding to a form region, a character string obtained by inserting the character string of the other handwritten character string region in the position of the correction object in the character string of the handwritten character string region where the correction object was detected.

Description

Image processing apparatus and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] A certain image forming apparatus detects a character string area of handwritten characters included in a read image obtained from a document in which a handwritten character string is entered in a form area by a user, (b) based on form data, identifies position information of the form area where the handwritten character string should be written, (c) identifies the character string area within the form area in the read image, and (d) extracts a character string from the character string area (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-29228

[0004] However, in the above-described image forming apparatus, when the user corrects a typo or the like when entering a handwritten character string and separately enters the handwritten character string to be inserted at the correction position adjacent to the original handwritten character string, the handwritten character string separately entered for the correction may not be appropriately reflected, and there is a possibility that the character string corresponding to the form area may not be appropriately extracted.

[0005] The present invention has been made in view of the above problems, and an object thereof is to obtain an image processing apparatus and an image processing method in which even when the user makes a correction when entering a handwritten character string and separately enters the handwritten character string to be inserted at the correction position adjacent to the original handwritten character string, the character string corresponding to the form area is appropriately extracted.

[0006] The image processing apparatus according to the present invention comprises a form area identification unit that identifies a form area in a target image based on form data, a handwritten string area detection unit that detects a handwritten string area in the form area, and a handwritten string extraction unit that extracts a handwritten string for the handwritten string area corresponding to the form area. When the handwritten string extraction unit detects a correction object obtained by adding to a handwritten character and then erasing the handwritten character within the handwritten string area, it determines whether (a) there is another handwritten string area in a predetermined replacement string detection range adjacent to the correction object, and (b) if there is another handwritten string area in the replacement string detection range, it inserts the string of the other handwritten string area into the string of the handwritten string area where the correction object was detected, and the resulting string is the handwritten string corresponding to the form area.

[0007] The image processing method according to the present invention comprises the steps of: identifying a form region in a target image based on form data; detecting a handwritten string region in the form region; and extracting a handwritten string for the handwritten string region corresponding to the form region. In the handwritten string extraction step, if a correction object obtained by adding to and then erasing a handwritten character is detected within the handwritten string region, (a) it is determined whether another handwritten string region exists in a predetermined replacement string detection range adjacent to the correction object; and (b) if another handwritten string region exists in the replacement string detection range, the string obtained by inserting the string of the other handwritten string region at the position of the correction object in the string of the handwritten string region where the correction object was detected is defined as the handwritten string corresponding to the form region.

[0008] The image processing program according to the present invention causes a computer to function as the form area identification unit, the handwritten character area detection unit, and the handwritten character extraction unit described above.

[0009] According to the present invention, an image processing device and image processing method are obtained that appropriately extract the string corresponding to the form area even when a user makes a correction while entering a handwritten string and separately enters the handwritten string to be inserted in the correction location adjacent to the original handwritten string.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram illustrating the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram illustrating the extraction of a string from a handwritten string area in a form area. Figure 3 is a diagram illustrating an example of a correction object in a handwritten string area. Figure 4 is a diagram illustrating the search for replacement candidates corresponding to a correction object (1 / 2). Figure 5 is a diagram illustrating the search for replacement candidates corresponding to a correction object (2 / 2). Figure 6 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0014] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as form data 2b) necessary for the processing described later. The image processing program is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2. The form data 2b is data that indicates the position and size of a form area within a specific target image where a handwritten string should be entered by the user.

[0016] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0017] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0018] In this case, the arithmetic processing unit 1 operates as the aforementioned processing units: the target image acquisition unit 11, the form area identification unit 12, the handwritten character area detection unit 13, and the handwritten character extraction unit 14.

[0019] The target image acquisition unit 11 acquires the target image (raster image data) from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM or the like.

[0020] For example, this target image is a scanned image of a given document that includes a form area where a user has entered handwritten text. In this case, the form area is a rectangular area enclosed by a grid.

[0021] The form area identification unit 12 identifies the form area (specifically, its position and size) in the target image based on the form data 2b.

[0022] The handwritten character area detection unit 13 detects handwritten character areas within the form area described above. For example, the handwritten character area detection unit 13 performs character recognition processing using a machine learning-trained learner and detects handwritten character areas and correction objects described later in the target image using an existing object detection method, and identifies the handwritten character areas within the form area from among the detected handwritten character areas. Alternatively, the handwritten character area detection unit 13 may perform the above-mentioned character recognition processing only on the portion of the target image within the form area to detect handwritten character areas within the form area. Furthermore, the handwritten character area detection unit 13 detects objects using the above-mentioned learner, classifies the detected objects into handwritten character objects, correction objects, and other objects, and detects the set of handwritten character objects as a handwritten character area.

[0023] Figure 2 illustrates the extraction of a string from a handwritten string area in a form area. The handwritten string extraction unit 14 extracts a handwritten string from the handwritten string area corresponding to the form area, for example, as shown in Figure 2. For example, the handwritten string extraction unit 14 obtains the text data of the handwritten string from the result of the character recognition process described above and stores it in association with the form area (for example, the identifier of the form area).

[0024] Figure 3 shows an example of a correction object in the handwritten text area. Figures 4 and 5 illustrate the search for replacement candidates corresponding to the correction object.

[0025] When the handwritten string extraction unit 14 detects a correction object within a handwritten string area, for example as shown in Figure 3, it determines whether another handwritten string area exists in a predetermined replacement string detection range adjacent to the correction object, for example as shown in Figures 4 and 5, and if another handwritten string area exists in the replacement string detection range (for example, in the case of Figure 4), it inserts the string of that other handwritten string area (in the case of Figure 4, "center") into the position of the correction object in the string of the handwritten string area where the correction object was detected, and uses the resulting string as the handwritten string corresponding to that form area.

[0026] A correction object is an object obtained by adding to handwritten text and then canceling it out, as shown in Figure 3, for example. For example, as shown in Figure 3, handwritten text can be canceled out by superimposing a strikethrough line over it (correction examples #1 and #2), or by blacking out (solid coloring) the handwritten text (correction example #3).

[0027] For example, as shown in Figure 3, when a user corrects an incorrect handwritten character, such as a typographical error, they add to the incorrect character to erase it, and then write the correct handwritten character corresponding to the erased character. In this case, the user generally writes the correct handwritten character on a separate line from the erased character (correction object) (either above or below the correction object (correction example #1)), or writes the correct handwritten character immediately following the erased character (correction object) along the direction of the text (correction examples #2, #3).

[0028] In this embodiment, as shown in Figures 4 and 5, for example, the replacement string detection range has the same range as the width of the correction object in the string direction, and in the direction perpendicular to the string direction, it has the range from a position equal to the height of the correction object from the top of the correction object to a position equal to the height of the correction object from the bottom of the correction object. Here, the string direction is the longitudinal direction of the handwritten string area. Then, as shown in Figure 4, for example, if the centroid of a certain handwritten string area among the detected handwritten string areas is within the replacement string detection range (for example, in the case of correction example #1), the handwritten string extraction unit 14 detects that handwritten string area as another handwritten string area (i.e., a replacement candidate).

[0029] Furthermore, as shown in Figure 5, for example, if the centroid of any of the detected handwritten string regions is not within the replacement string detection range, the handwritten string extraction unit 14 does not perform the above-described replacement process (i.e., insertion of replacement candidate strings) because there are no replacement candidates (for example, in correction examples #2 and #3).

[0030] Here, if there are multiple handwritten string regions other than the handwritten string region containing the correction object within the replacement string detection range, the handwritten string extraction unit 14 (a) selects from among those other handwritten string regions the handwritten string region having the width closest to the width (i.e., length in the string direction) of the correction object. In this case, the handwritten string extraction unit 14 inserts the string of the selected handwritten string region into the string of the handwritten string region where the correction object was detected, and uses the resulting string as the handwritten string corresponding to the form region.

[0031] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 6 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0032] When the target image acquisition unit 11 acquires a target image (step S1), the form area identification unit 12 identifies the form area in the target image based on the form data 2b corresponding to the target image (step S2), and the handwritten character area detection unit 13 detects the handwritten character area in the form area (step S3).

[0033] The handwritten string extraction unit 14 determines whether or not there is a correction object within the detected handwritten string area (step S4).

[0034] If no correction object is detected within the detected handwritten string area, the handwritten string extraction unit 14 will ignore the correction object and extract the handwritten string for the handwritten string area corresponding to the form area without performing a replacement process (i.e., inserting the correct string as described above) (step S5).

[0035] On the other hand, if a correction object is detected within the detected handwritten string area, the handwritten string extraction unit 14 determines whether there is at least one handwritten string area adjacent to the handwritten string area containing the correction object (step S6). If there is an adjacent handwritten string area, the longest handwritten string area among the handwritten string area containing the correction object and the adjacent handwritten string area is designated as the handwritten string area to be replaced (usually the handwritten string area containing the correction object) (step S7).

[0036] Furthermore, if there is no handwritten string area adjacent to the handwritten string area containing the correction object, the handwritten string extraction unit 14 ignores the correction object and extracts the handwritten string for the handwritten string area corresponding to the form area, since the correct handwritten string is not included (for example, in correction examples #2 and #3). (Step S5)

[0037] Then, the handwritten string extraction unit 14 attempts to detect replacement candidates for the correction object as described above, and determines whether or not at least one replacement candidate for the correction object has been detected (step S8).

[0038] If no replacement targets are detected, the handwritten string extraction unit 14 ignores the correction objects and extracts the handwritten string for the handwritten string area corresponding to the form area (step S5).

[0039] On the other hand, if at least one replacement candidate is detected, the handwritten string extraction unit 14 determines whether multiple replacement candidates have been detected for the correction object (step S9). If multiple replacement candidates are detected, as described above, it selects one replacement candidate from the multiple detected replacement candidates (step S10), and generates a handwritten string for the form area by inserting the selected replacement candidate string from the handwritten string area at the position of the correction object in the string of the handwritten string area to be replaced (step S11).

[0040] As described above, according to the above embodiment, the form area identification unit 12 identifies the form area in the target image based on the form data. The handwritten string area detection unit 13 detects the handwritten string area in the form area. The handwritten string extraction unit 14 extracts the handwritten string for the handwritten string area corresponding to the form area. When the handwritten string extraction unit 14 detects a correction object within the handwritten string area, it determines whether (a) another handwritten string area exists in a predetermined replacement string detection range adjacent to the correction object, and (b) if another handwritten string area exists in the replacement string detection range, it inserts the string of that other handwritten string area into the position of the correction object in the string of the handwritten string area where the correction object was detected, and the resulting string is considered the handwritten string corresponding to the form area.

[0041] This ensures that even if a user makes corrections to handwritten text and inserts the corrected text adjacent to the original handwritten text, the corresponding text in the form area will be properly extracted.

[0042] Various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, such changes and modifications are intended to be included within the scope of the claims.

[0043] For example, in the above embodiment, in the replacement process, the character string (text data) in the handwritten character string area of the replacement candidate is inserted at the position of the correction object in the character string (text data) of the handwritten character string area to be replaced. However, in the replacement process, the image of the handwritten character string area of the replacement candidate may be used to replace the correction object in the handwritten character string area to be replaced, and the above-described character recognition process may be performed on the handwritten character string area to be replaced after replacement to identify the corresponding handwritten character string (text data).

[0044] Further, in the above embodiment, when the correction object is at the end (front end or rear end) of the character string in the handwritten character string area, the replacement character string detection range may be widened by a predetermined width beyond the end in the character string direction. By doing so, even if the correct character string is written diagonally up or diagonally down, it can be appropriately detected.

[0045] Further, in the above embodiment, the image processing apparatus is a device directly used by the user, but it may also be a device such as a server (e.g., a cloud server) indirectly used by the user via a network. In that case, the user interface of the client device is used instead of the display device 4 and the input device 5.

[0046] The present invention is applicable, for example, to an image processing apparatus.

Claims

1. An image processing apparatus comprising: a form area identification unit that identifies a form area in a target image based on form data; a handwritten string area detection unit that detects a handwritten string area in the form area; and a handwritten string extraction unit that extracts a handwritten string for the handwritten string area corresponding to the form area, wherein when the handwritten string extraction unit detects a correction object obtained by adding to a handwritten character and then erasing the handwritten character within the handwritten string area, it determines whether or not another handwritten string area exists in a predetermined replacement string detection range adjacent to the correction object, and if another handwritten string area exists in the replacement string detection range, it inserts the string of the other handwritten string area into the string of the handwritten string area where the correction object was detected, and makes the resulting string the handwritten string corresponding to the form area.

2. The image processing apparatus according to claim 1, characterized in that the replacement string detection range has the same range as the width of the correction object in the string direction, and in a direction perpendicular to the string direction, it has a range from a position the same height as the height of the correction object from the upper end of the correction object to a position the same height as the height of the correction object from the lower end of the correction object, the string direction is the longitudinal direction of the handwritten string area, and the handwritten string extraction unit detects the handwritten string area whose centroid is within the replacement string detection range as the other handwritten string area when the centroid of the handwritten string area among the detected handwritten string areas is within the replacement string detection range.

3. The image processing apparatus according to claim 1, characterized in that, if there are multiple handwritten string regions other than the handwritten string region in the replacement string detection range, (a) select a handwritten string region having the width closest to the width of the correction object from among the other handwritten string regions, and (b) insert the string of the selected handwritten string region into the string of the handwritten string region where the correction object was detected, and make the resulting string the handwritten string corresponding to the form region.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the handwritten character region detection unit detects the handwritten character region and the correction object using a machine learning-trained learner.

5. An image processing method comprising: a step of identifying a form area in a target image based on form data; a step of detecting a handwritten string area in the form area; and a handwritten string extraction step of extracting a handwritten string for the handwritten string area corresponding to the form area, wherein in the handwritten string extraction step, if a correction object obtained by adding to a handwritten character and then erasing the handwritten character is detected within the handwritten string area, (a) it is determined whether or not another handwritten string area exists in a predetermined replacement string detection range adjacent to the correction object, and (b) if another handwritten string area exists in the replacement string detection range, the string obtained by inserting the string of the other handwritten string area at the position of the correction object in the string of the handwritten string area where the correction object was detected is made to be the handwritten string corresponding to the form area.

Images