Image processing device and image processing method

Abstract

If registered data (2b) does not include a name matching a name indicated by a handwritten character string in a document image of a document in a specific format, a personal identification processing unit (16) determines whether the registered data (2b) includes image information matching image information pertaining to a person discernment image detected by a person discernment image detection unit (14). If the registered data (2b) includes image information matching the image information pertaining to the person discernment image detected by the person discernment image detection unit (14), the personal identification processing unit (16) determines that the aforementioned document belongs to a person corresponding to the image information in the registered data (2b).

Description

Image processing apparatus and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] A certain image forming apparatus detects a character string area of handwritten characters included in a read image obtained from a document in which a handwritten character string is entered in a form area by a user, (b) specifies position information of the form area where the handwritten character string should be described based on form data, (c) specifies the character string area within the form area in the read image, and (d) extracts a character string from the character string area (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-29228

[0004] When obtaining text data of a name by image reading and character recognition from a document (form) in which an individual has handwritten their name in a predetermined name field, due to a change in the name such as marriage, the text data of the name obtained this time does not match the text data of the name obtained from a previous document, and it is determined that the person is a different person from the person whose name was obtained from the previous document.

[0005] The present invention has been made in view of the above problems, and when identifying an individual from a document image of a document (form) in which an individual has handwritten their name in a predetermined name field, even if the name or name notation changes, an image processing apparatus and an image processing method that appropriately determine that the person in the document image is the person with the name obtained in advance are obtained.

[0006] The image processing apparatus according to the present invention comprises: a character detection unit that detects typed and handwritten character strings in a document image of a document of a specific format; a character recognition processing unit that obtains text data of the handwritten character strings through character recognition processing; a personal identification image detection unit that detects a personal identification image in the document image; and, if no name matching the name as a handwritten character string exists in the registered data, (a) a determination is made whether or not there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit; and (b) if there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit, a determination is made that the document belongs to the person in the registered data; and if there is no image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit, a determination is made that the document does not belong to the person in the registered data.

[0007] The image processing method according to the present invention comprises: a character detection step for detecting typed and handwritten characters in a document image of a document of a specific format; a character recognition processing step for obtaining text data of the handwritten characters by character recognition processing; a personal identification image detection step for detecting a personal identification image in the document image; and, if no name matching the name as a handwritten character exists in the registered data, (a) determining whether there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step; and (b) determining that if there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step, the document belongs to the person in the registered data; and if there is no image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step, the document does not belong to the person in the registered data.

[0008] The image processing program according to the present invention causes a computer to function as the above-mentioned string detection unit, character recognition processing unit, personal identification image detection unit, and personal identification processing unit.

[0009] According to the present invention, when identifying an individual from a document image of a document (form) in which an individual has handwritten their name in a designated name field, an image processing device and image processing method are provided that can appropriately determine that the person in the document image is the person with the previously obtained name, even if the name or its spelling changes.

[0010] The above or other objects, features, and advantages of the present invention will become even more apparent from the following detailed description in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a diagram showing an example of a document image. Figure 3 is a flowchart illustrating the operation of the image processing apparatus shown in Figure 1.

[0012] Embodiments of the present invention will be described below with reference to the figures.

[0013] Figure 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Figure 1 is an information processing apparatus such as a personal computer or a server, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction printer, etc.), and comprises a processing unit 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, and the like.

[0014] The arithmetic processing unit 1 includes a computer and operates as various processing units by executing image processing programs on that computer. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and operates as a predetermined processing unit by loading programs stored in the ROM or storage device 2 into the RAM and executing them with the CPU. The arithmetic processing unit 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 2 is a non-volatile storage device such as flash memory, and stores the image processing program 2a and data (such as registration data 2b) necessary for the processing described later. The image processing program is stored, for example, on a non-temporary, computer-readable recording medium and installed from that recording medium to the storage device 2. The registration data 2b includes the name (text data) and image information (such as image data for personal identification) for each person.

[0016] Communication device 3 is a device that communicates data with external devices, such as a network interface or peripheral device interface. Display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. Input device 5 is a device that detects user operations, such as a keyboard or touch panel.

[0017] The internal device 6 is a device that performs a predetermined function of the image processing device. For example, if the image processing device is an image forming apparatus, the internal device 6 may be an image reading device that optically reads an image from a document, or a printing device that prints an image onto printing paper.

[0018] In this configuration, the arithmetic processing unit 1 operates as the aforementioned processing units: the target image acquisition unit 11, the character detection unit 12, the character recognition processing unit 13, the personal identification image detection unit 14, the image correction processing unit 15, and the personal identification processing unit 16.

[0019] The target image acquisition unit 11 acquires the target image (raster image data) as a document image from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM or the like.

[0020] Figure 2 shows an example of a document image. For example, as shown in Figure 2, this document image is a read image of a document in a specific format in which a person has written their name as handwritten text and an image of that person's personal identification has been added. Here, the personal identification image includes a facial photograph or a thumbprint. Here, the document in a specific format is a document such as an application form that is created periodically (e.g., a driver's license renewal application form, a general passport issuance application form). The document image shown in Figure 2 is a driver's license renewal application form with a facial photograph added as the personal identification image.

[0021] The character detection unit 12 detects typed and handwritten characters in the document image using a machine learning model trained using, for example, an existing method. Specifically, the character detection unit 12 detects character strings in the document image and classifies the detected character strings into typed and handwritten characters.

[0022] The character recognition processing unit 13 obtains handwritten string text data using existing character recognition processing.

[0023] The personal identification image detection unit 14 detects the aforementioned personal identification image in the document image.

[0024] The image correction processing unit 15 performs image correction on the aforementioned personal identification images so that the image information of the same person matches each other.

[0025] If no name matching the handwritten string name described above exists in the registered data 2b, the personal identification processing unit 16 determines whether (a) there is image information in the registered data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, and (b) if there is image information in the registered data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, it determines that the document image (i.e., the document in the document image) belongs to the person in the image information in the registered data 2b, and if there is no image information in the registered data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, it determines that the document image (i.e., the document in the document image) does not belong to the person in any of the image information in the registered data 2b.

[0026] If the above-mentioned image correction has been performed, the personal identification processing unit 16 obtains image information from the personal identification image after image correction.

[0027] Furthermore, if the personal identification image includes a facial photograph or a thumbprint, the personal identification processing unit 16 obtains image information from the facial photograph or thumbprint.

[0028] The image information may be raster image data of face images or thumbprint images within the personal identification image, or it may be predetermined features of the face images or thumbprint images. Furthermore, whether the image information matches or not is determined based on the difference between the features in the case of features, and in the case of face images, it is determined using, for example, a machine learning model trained using an existing method.

[0029] Furthermore, as an image correction measure, if the personal identification processing unit 16 includes a facial photograph, it resizes the personal identification image or the facial photograph in the personal identification image so that the size of the facial outline in the facial photograph becomes a specific size.

[0030] Furthermore, as an image correction measure, the personal identification processing unit 16 rotates the face image or personal identification image to reduce the tilt of the face image if the face image is tilted when the personal identification image includes a face photograph. The tilt of the face image may be determined from the positions of the eyes, nose, mouth, etc. in the face image, or it may be obtained by a machine learning model that has been trained using an existing method.

[0031] Furthermore, as part of image correction, the personal identification processing unit 16 rotates the personal identification image to reduce its tilt when the personal identification image is a rectangular photographic image and is tilted. The tilt of the personal identification image is detected by detecting the edges of the rectangular shape and using the tilt of the edges as the tilt of the personal identification image.

[0032] Furthermore, in this embodiment, if there is no name in the registered data 2b that matches the name as a handwritten string as described above, and there is no image information in the registered data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, the personal identification processing unit 16 associates the image information of the personal identification image detected by the personal identification image detection unit 14 with the name as a handwritten string as described above and registers it in the registered data 2b.

[0033] Furthermore, in this embodiment, if there is no name in the registered data that matches the name as a handwritten string as described above, the personal identification processing unit 16 outputs a message prompting the user to update the name in the registered data 2b to the name as a handwritten string if there is image information in the registered data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14. For example, this message is displayed on the display device 4.

[0034] Next, the operation of the image processing device shown in Figure 1 will be explained. Figure 3 is a flowchart illustrating the operation of the image processing device shown in Figure 1.

[0035] When the target image acquisition unit 11 acquires a document image (step S1), the character detection unit 12 detects character strings in the document image (step S2), extracts handwritten names from the detected character strings, and the character recognition processing unit 13 performs character recognition processing on these character strings (step S3).

[0036] Furthermore, the personal identification image detection unit 14 detects the aforementioned personal identification images in the document image (step S4), and the image correction processing unit 15 performs image correction on the aforementioned personal identification images so that the image information of the same person matches each other (step S5).

[0037] The personal identification processing unit 16 first determines whether a name (text data) matching the name (text data) obtained from the document image is registered in the registration data 2b as the name of any person (step S6).

[0038] If no name matching the name obtained from the document image is registered in the registration data 2b as the name of any person, the personal identification processing unit 16 determines whether or not image information matching the image information of the personal identification image obtained from the document image is registered in the registration data 2b as the image information of any person (step S7).

[0039] If no image information matching the image information of the personal identification image obtained from the document image is registered in the registration data 2b as image information of any person, the personal identification processing unit 16 registers the name (text data) and the image information of the personal identification image obtained from the document image in the registration data 2b (step S8). Subsequently, the personal identification processing unit 16 outputs the name, etc. obtained from the document image (step S9). For example, the name, etc. obtained from the document image may be provided to a later processing step or stored in the storage device 2.

[0040] On the other hand, in step S7, if the personal identification processing unit 16 determines that image information matching the image information of the personal identification image obtained from the document image is registered in the registration data 2b as the image information of one of the individuals, the personal identification processing unit 16 displays a message indicating that although the image information matches, the name does not match, and therefore the name in the registration data 2b should be updated (step S10). Subsequently, the personal identification processing unit 16 determines whether or not to update the name in the registration data 2b according to the user operation to the input device 5 (step S11), and if it determines that the name in the registration data 2b should be updated, it updates the name in the registration data 2b with the name obtained from the document image (step S12). After that, the personal identification processing unit 16 outputs the name obtained from the document image (or the updated name if updated) (step S9).

[0041] On the other hand, in step S6, when it is determined that the name obtained from the document image matches the name of any person registered in the registration data 2b, the personal identification processing unit 16 outputs the name and the like obtained from the document image (step S9).

[0042] As described above, according to the above embodiment, the character string detection unit 12 detects the printed character string and the handwritten character string in the document image of the document in a specific format. The character recognition processing unit 13 obtains the text data of the handwritten character string by character recognition processing. The personal identification image detection unit 14 detects the personal identification image in the document image. When the name that matches the name as the handwritten character string is not in the registration data 2b, the personal identification processing unit 16 determines (a) whether there is image information in the registration data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, and (b) if there is image information in the registration data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, it is determined that the above document belongs to the person of the image information in the registration data 2b, and if there is no image information in the registration data 2b that matches the image information of the personal identification image detected by the personal identification image detection unit 14, it is determined that the above document does not belong to the person of the image information in the registration data 2b.

[0043] Thereby, based on the personal identification image added to the document, the person of the document is identified. Therefore, even if the name or the name notation changes, it is appropriately determined that the person of the document image is the person with the previously obtained name.

[0044] Various changes and modifications to the above embodiment will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, it is intended that such changes and modifications be included in the scope of the claims.

[0045] For example, in the above embodiment, the above image information may be other biometric information.

[0046] Also, in the above embodiment, the above personal identification image may be a seal impression.

[0047] Furthermore, in the above embodiment, the image processing device is a device used directly by the user, but it may also be a device such as a server (e.g., a cloud server) used indirectly by the user via a network. In that case, the user interface of the client device is used instead of the display device 4 and the input device 5.

[0048] The present invention is applicable, for example, to an image processing apparatus.

Claims

1. An image processing device comprising: a character detection unit that detects typed and handwritten characters in a document image of a document of a specific format; a character recognition processing unit that obtains text data of the handwritten characters through character recognition processing; a personal identification image detection unit that detects a personal identification image in the document image; and, if no name matching the name as a handwritten character exists in the registered data, (a) a determination is made whether or not there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit; and (b) if there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit, a determination is made that the document belongs to the person in the registered data; and if there is no image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit, a determination is made that the document does not belong to the person in the registered data.

2. The image processing apparatus according to claim 1, wherein the personal identification image includes a facial photograph or a thumbprint, and the personal identification processing unit obtains the image information from the facial photograph or the thumbprint.

3. The image processing apparatus according to claim 1, further comprising an image correction unit that corrects the personal identification images so that the image information of the same person matches each other, wherein the personal identification processing unit obtains the image information from the personal identification images after the image correction.

4. The image processing apparatus according to claim 3, wherein the personal identification processing unit changes the size of the personal identification image or the face photograph in the personal identification image so that the size of the face outline in the face photograph becomes a specific size when the personal identification image includes a face photograph.

5. The image processing apparatus according to claim 3, wherein, if the personal identification processing unit includes a photograph of a face, and the face image is tilted, the unit rotates the face image or the personal identification image to reduce the tilt of the face image.

6. The image processing apparatus according to any one of claims 3 to 5, characterized in that, when the personal identification image is a rectangular photographic image, the personal identification processing apparatus rotates the personal identification image to reduce the tilt of the personal identification image.

7. The image processing apparatus according to claim 1, wherein, if there is no name in the registered data that matches the name as a handwritten string, and there is no image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit, the personal identification processing unit associates the image information of the personal identification image detected by the personal identification image detection unit with the name as a handwritten string and registers them in the registered data.

8. The image processing apparatus according to claim 1, wherein, if there is no name in the registered data that matches the name as a handwritten string, the personal identification processing unit outputs a message prompting the user to update the name in the registered data to the name as a handwritten string if there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection unit.

9. An image processing method comprising: a character detection step for detecting typed and handwritten characters in a document image of a document of a specific format; a character recognition processing step for obtaining text data of the handwritten characters by character recognition processing; a personal identification image detection step for detecting a personal identification image in the document image; and, if no name matching the name as a handwritten character exists in the registered data, (a) determining whether or not there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step; and (b) determining that if there is image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step, the document belongs to the person in the registered data; and if there is no image information in the registered data that matches the image information of the personal identification image detected by the personal identification image detection step, the document does not belong to the person in the registered data.

Images