Estimation device, learning device, estimation method, learning method, and recording medium

Abstract

An estimation device according to the present disclosure has an estimated heat map generation unit and a feature point information extraction unit. The estimated heat map generation unit estimates, from a fingerprint image, a first estimated heat map representing likelihoods of positions of fingerprint feature points and directions of fingerprint ridge lines extending from the fingerprint feature points. The feature point information extraction unit extracts, from the first estimated heat map, feature point information indicating the positions of fingerprint feature points and the directions of fingerprint ridge lines extending from the fingerprint feature points.

Description

Estimation Apparatus, Learning Apparatus, Estimation Method, Learning Method, and Recording Medium

[0001] The present disclosure relates to an estimation apparatus, a learning apparatus, an estimation method, a learning method, and a recording medium.

[0002] Related art is disclosed in Patent Document 1. Patent Document 1 refers to a facial feature point detection technique using a heatmap. The facial feature point detection technique receives a face image as input and outputs, as a heatmap, the existence probability of each feature point. The facial feature point detection technique outputs heatmaps corresponding to the number of feature points.

[0003] International Publication No. 2022 / 003982

[0004] An example object of the present disclosure is to develop a technique for extracting fingerprint feature point information from a fingerprint image.

[0005] According to one aspect of the present disclosure, there is provided an estimation apparatus comprising: estimated heatmap generation means for estimating, from a fingerprint image, a first estimated heatmap representing likelihood of a position of a fingerprint feature point and a direction of a fingerprint line extending from the fingerprint feature point; and feature point information extraction means for extracting feature point information indicating the position of the fingerprint feature point and the direction of the fingerprint line from the first estimated heatmap.

[0006] According to another aspect of the present disclosure, there is provided an estimation method performed by at least one computer, the method comprising: estimating, from a fingerprint image, a first estimated heatmap representing likelihood of a position of a fingerprint feature point and a direction of a fingerprint line extending from the fingerprint feature point; and extracting feature point information indicating the position of the fingerprint feature point and the direction of the fingerprint line from the first estimated heatmap.

[0007] According to another aspect of the present disclosure, there is provided a recording medium having recorded thereon a program that causes a computer to function as: estimated heatmap generation means for estimating, from a fingerprint image, a first estimated heatmap representing likelihood of a position of a fingerprint feature point and a direction of a fingerprint line extending from the fingerprint feature point; and feature point information extraction means for extracting feature point information indicating the position of the fingerprint feature point and the direction of the fingerprint line from the first estimated heatmap.

[0008] According to one aspect of the present disclosure, there is provided a learning device comprising: estimated heat map generation means for estimating, from a fingerprint image, a first estimated heat map representing likelihood of a position of a fingerprint minutia and a direction of a fingerprint ridge extending from the fingerprint minutia; correct heat map calculation means for generating, from correct fingerprint minutia information, a first correct heat map representing likelihood of the position of the fingerprint minutia and the direction of the fingerprint ridge extending from the fingerprint minutia; and first learning means for updating parameters of a model that estimates the first estimated heat map based on the first estimated heat map and the first correct heat map.

[0009] According to one aspect of the present disclosure, there is provided a learning device comprising: minutia information estimation means for estimating estimated fingerprint minutia information from a fingerprint image; matching means for outputting a matching score between first estimated fingerprint minutia information estimated from a first fingerprint image and second estimated fingerprint minutia information estimated from a second fingerprint image different from the first fingerprint image; and second learning means for updating parameters of a model that estimates the estimated fingerprint minutia information and parameters of a model that outputs the matching score based on the matching score and a correct answer indicating whether the first fingerprint image and the second fingerprint image are fingerprint images capturing the same fingerprint.

[0010] According to one aspect of the present disclosure, there is provided a learning method in which at least one computer: estimates, from a fingerprint image, a first estimated heat map representing likelihood of a position of a fingerprint minutia and a direction of a fingerprint ridge extending from the fingerprint minutia; generates, from correct fingerprint minutia information, a first correct heat map representing likelihood of the position of the fingerprint minutia and the direction of the fingerprint ridge extending from the fingerprint minutia; and updates parameters of a model that estimates the first estimated heat map based on the first estimated heat map and the first correct heat map.

[0011] Furthermore, according to one aspect of this disclosure, a recording medium is provided which records a program that causes a computer to function as: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from a fingerprint image; a ground truth heatmap calculation means for generating a first ground truth heatmap representing the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points from ground truth fingerprint feature point information; and a first learning means for updating the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and the first ground truth heatmap.

[0012] According to this example of disclosure, it is possible to develop techniques for extracting feature point information of fingerprints from fingerprint images.

[0013] Figure 1 is a diagram showing an example of a functional block diagram of an estimation device. Figure 2 is a flowchart showing an example of the processing flow of an estimation device. Figure 3 is a diagram illustrating an example of the processing performed by the estimation device. Figure 4 is a diagram showing an example of the hardware configuration of an estimation device and a learning device. Figure 5 is a diagram illustrating an example of feature point information. Figure 6 is a flowchart showing another example of the processing flow of an estimation device. Figure 7 is a diagram showing an example of a functional block diagram of a learning device. Figure 8 is a diagram illustrating an example of the processing performed by a learning device. Figure 9 is a diagram illustrating another example of the processing flow performed by a learning device. Figure 10 is a flowchart showing an example of the processing flow of a learning device. Figure 11 is a diagram showing another example of a functional block diagram of a learning device. Figure 12 is a flowchart showing another example of the processing flow of a learning device. Figure 13 is a diagram showing another example of a functional block diagram of a learning device.

[0014] The embodiments of this disclosure will be described below with reference to the drawings. In this disclosure, the drawings are associated with one or more embodiments. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted where appropriate.

[0015] <<First Embodiment>> Figure 1 is a functional block diagram showing an overview of the estimation device 10. Figure 2 is a flowchart showing an example of the processing flow performed by the estimation device 10.

[0016] As shown in Figure 1, the estimation device 10 has a feature point information estimation unit 11. The feature point information estimation unit 11 has an estimation heatmap generation unit 111 and a feature point information extraction unit 112. These functional units execute the process shown in the flowchart of Figure 2.

[0017] In S10, the estimated heatmap generation unit 111 estimates a first estimated heatmap from the fingerprint image that represents the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points. In S11, the feature point information extraction unit 112 extracts feature point information from the first estimated heatmap that indicates the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points.

[0018] Incidentally, some fingerprint images may be of poor quality due to noise, smudges, etc. For example, residual fingerprints can be such low-quality fingerprint images. Simply extracting the features that are clearly visible in the fingerprint image will not be enough to extract a sufficient number of features from such low-quality fingerprint images, making accurate matching impossible.

[0019] Therefore, the estimation device 10 estimates a "first estimated heatmap" from the fingerprint image, which represents the "likelihood" of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points. The estimation device 10 then extracts feature point information from this "first estimated heatmap". With such an estimation device 10, it is possible to estimate and extract features that are not clearly visible in the fingerprint image. As a result, a sufficient amount of features can be extracted from low-quality fingerprint images, enabling matching with sufficient accuracy.

[0020] Furthermore, in fingerprint matching, matching only the "location of fingerprint feature points" is insufficient to achieve adequate accuracy; matching also requires the use of the "direction of fingerprint lines extending from the fingerprint feature points." Therefore, in the process of extracting feature point information from fingerprint images, it is desirable to extract not only the "location of fingerprint feature points" but also the "direction of fingerprint lines extending from the fingerprint feature points" with sufficient accuracy.

[0021] The estimation device 10 is configured to solve these problems. As described above, the estimation device 10 estimates a first estimation heatmap from the fingerprint image that represents the likelihood of the "position of fingerprint feature points" and the "direction of fingerprint lines extending from the fingerprint feature points." The estimation device 10 then extracts feature point information indicating the "position of fingerprint feature points" and the "direction of fingerprint lines extending from the fingerprint feature points" from the first estimation heatmap. With such an estimation device 10, the "position of fingerprint feature points" and the "direction of fingerprint lines extending from the fingerprint feature points" can be estimated with high accuracy.

[0022] Incidentally, Patent Document 1 mentioned above refers to a "face" feature point detection technology that uses a heat map. This face feature point detection technology uses a heat map to detect the "position of face feature points" and does not include a configuration for estimating the "direction of lines extending from face feature points." In fact, the concept of "lines extending from face feature points" does not exist. Therefore, if this face feature point detection technology were applied to extracting "fingerprint" feature points, it would only result in a configuration that estimates the "position" of fingerprint feature points using a heat map, and would not result in a configuration that estimates the "direction of lines extending from fingerprint feature points."

[0023] As explained above, the estimation device 10 makes it possible to develop the technology for extracting fingerprint feature point information from fingerprint images.

[0024] <<Second Embodiment>> <Overview> The estimation device 10 of the second embodiment is a concrete implementation of the configuration of the estimation device 10 of the first embodiment. As shown in Figure 3, the estimation device 10 generates an estimated heatmap from a fingerprint image and extracts fingerprint feature point information from the estimated heatmap. A detailed explanation follows below.

[0025] <Hardware Configuration> First, an example of the hardware configuration of the estimation device 10 will be described. Each functional part of the estimation device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various modifications to the implementation method and the device. The software includes programs that are stored in the device from the time of shipment, as well as programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0026] Figure 4 is a block diagram illustrating the hardware configuration of the estimation device 10. As shown in Figure 4, the estimation device 10 includes a processor 1A, memory 2A, input / output interface 3A, peripheral circuitry 4A, and bus 5A. The peripheral circuitry 4A includes various modules. The processing unit 10 does not necessarily have peripheral circuitry 4A. The estimation device 10 may also be composed of multiple physically and / or logically separated devices. In this case, each of the multiple devices may have the above hardware configuration.

[0027] Bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to send and receive data to and from each other. The processor 1A is a processing unit such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit). Memory 2A is a memory such as RAM (Random Access Memory) or ROM (Read Only Memory). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes interfaces for connecting to a communication network such as the Internet. Input devices include, for example, a keyboard, mouse, microphone, physical buttons, touch panel, etc. Output devices include, for example, a display, projection device, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on their calculation results.

[0028] <Fingerprint Feature Point Information> First, we will explain the fingerprint feature point information handled by the estimation device 10.

[0029] "Feature point information" refers to information about fingerprint feature points detected from a fingerprint. By using this feature point information for matching, it is possible to determine whether fingerprints match or not.

[0030] A "fingerprint feature point" includes, for example, at least one of the endpoints shown in Figure 5(A) and the branching points shown in Figure 5(B). An endpoint is a point that marks the end of a fingerprint line. A branching point is a point where a fingerprint line branches.

[0031] "Fingerprint lines" are lines that make up a fingerprint and include at least one of ridges and valleys.

[0032] Feature point information includes positional and directional components of fingerprint feature points. Feature point information may further include type components of fingerprint feature points.

[0033] The positional component indicates the location of the fingerprint feature points. The location of the fingerprint feature points is indicated, for example, by the coordinates of the image coordinate system set for the fingerprint image. The positional component indicates the relative positional relationships of multiple fingerprint feature points.

[0034] The directional component indicates the direction of the fingerprint lines extending from the fingerprint feature points. The direction of the fingerprint lines is indicated by the angle θ between the fingerprint line extending from the endpoint (fingerprint feature point) and the reference line S, as shown in Figure 5(A), for example. Alternatively, the direction of the fingerprint lines is indicated by the angle θ between the direction of the branch extending from the branch point (fingerprint feature point) and the reference line S, as shown in Figure 5(B), for example. The direction of the branch is, for example, the direction midway between the directions in which the two branched fingerprint lines extend. The reference line S may be defined, for example, as a line parallel to the x-axis direction of the image coordinate system set for the fingerprint image (e.g., the horizontal direction of the fingerprint image), or it may be defined in other ways.

[0035] The type component indicates the type of fingerprint feature point. The type of fingerprint feature point (feature point type) includes one of the following: delta, core, endpoint, and branch point. Fingerprint feature points corresponding to delta are located inside or near the triangular shape formed by the fingerprint lines. Fingerprint feature points corresponding to core are located inside or near the curves such as vortices, circles, and arches formed by the fingerprint lines.

[0036] <Functional Configuration> Next, the functional configuration of the estimation device 10 will be described in detail. Figure 1 is an example of a functional block diagram of the estimation device 10. As shown in the figure, the estimation device 10 has a feature point information estimation unit 11. The feature point information estimation unit 11 has an estimation heatmap generation unit 111 and a feature point information extraction unit 112.

[0037] The estimated heatmap generation unit 111 estimates a first estimated heatmap from the fingerprint image, which represents the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points.

[0038] A "fingerprint image" is an image that shows a fingerprint (an image that records a fingerprint). A fingerprint image is an image that records the surface pattern of the fingertip, acquired by ink or an optical sensor.

[0039] The "first estimated heatmap" represents the likelihood of the location of fingerprint feature points and the direction of fingerprint lines extending from those feature points. In other words, the first estimated heatmap represents the likelihood that fingerprint feature points exist at each location in the fingerprint image, extending in each direction. Each location in the fingerprint image is represented, for example, by coordinates in the image coordinate system set for the fingerprint image. Each location in the fingerprint image may also correspond to each pixel in the fingerprint image. Hereafter, "coordinates in the image coordinate system set for the fingerprint image" may simply be referred to as "coordinates." Also, "each location in the fingerprint image" may be referred to as "each coordinate in the fingerprint image."

[0040] The estimated heatmap generation unit 111 can estimate a plurality of first estimated heatmaps corresponding to each of a plurality of predefined directions. The first estimated heatmap corresponding to the first direction among the plurality of directions represents the likelihood that a fingerprint feature point exists at each coordinate in the fingerprint image that is the starting point of a fingerprint line extending in the first direction.

[0041] Multiple directions are defined in advance. For example, they may be defined in 60° increments, such as 0° (360°), 60°, 120°, 180°, 240°, and 300°. In this case, six directions are defined in advance. The estimated heatmap generation unit 111 can then estimate six first estimated heatmaps, each corresponding to one of the six directions. For example, the first estimated heatmap corresponding to 60° represents the likelihood that a fingerprint feature point exists at each coordinate in the fingerprint image that is the starting point of a fingerprint line extending in the 60° direction (a fingerprint line with θ = 60°).

[0042] Note that the 60° increment is merely an example and is not limited to it. Multiple directions may be defined using other numerical intervals, such as 10° or 30° increments. The smaller the numerical interval for multiple directions, the more first estimated heatmaps the estimated heatmap generation unit 111 generates, increasing the processing load on the computer. On the other hand, the smaller the numerical interval for multiple directions, the higher the accuracy of estimating the direction of fingerprint lines extending from fingerprint feature points based on the first estimated heatmap.

[0043] The estimated heatmap generation unit 111 is composed of a trained neural network having a structure in which, for example, an encoder and a decoder are coupled in series. The configuration of such an estimated heatmap generation unit 111 is constructed using widely known techniques. The trained model takes a fingerprint image as input and outputs multiple first estimated heatmaps corresponding to each of multiple directions. The characteristic training method of the trained model will be described in the following embodiment.

[0044] The feature point information extraction unit 112 extracts feature point information from the first estimated heatmap, indicating the location of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points. The feature point information is as explained with reference to Figure 5.

[0045] The feature point information extraction unit 112 can estimate the location of a fingerprint feature point and the direction of fingerprint lines extending from the fingerprint feature point, based on the "likelihood of the existence of fingerprint feature points extending in each direction at each position in the fingerprint image" shown in the first estimated heatmap. For example, if the "likelihood of the existence of a fingerprint feature point extending in a first direction at a first position in the fingerprint image" satisfies a predetermined condition, the feature point information extraction unit 112 can estimate that a fingerprint feature point extending in a first direction exists at a first position in the fingerprint image. The feature point information extraction unit 112 can then generate feature point information that shows such an estimation result. The following describes a specific example of the process that realizes such estimation, but it is not limited to this example.

[0046] First, we will explain the process of estimating the coordinates where fingerprint feature points exist based on the first estimated heatmap.

[0047] In one example, the feature point information extraction unit 112 finds the coordinates that take the local maximum value from the first estimated heatmap. Then, the feature point information extraction unit 112 compares the heatmap value (likelihood) at the found coordinates with a predefined threshold. If the heatmap value (likelihood) at the found coordinates exceeds the threshold, the feature point information extraction unit 112 estimates those coordinates as the coordinates where fingerprint feature points exist.

[0048] This threshold may vary depending on the type of fingerprint image. Poor quality fingerprint images (e.g., images of residual fingerprints) tend to have lower heatmap values. Conversely, high-quality fingerprint images tend to have higher heatmap values. Therefore, by setting an appropriate threshold according to the type of fingerprint image, an appropriate number of coordinates can be extracted as the positions of fingerprint feature points.

[0049] In one example, thresholds are defined in advance for each type of fingerprint image and registered in the estimation device 10. The operator inputs information indicating the type of fingerprint image to the estimation device 10. The feature point information extraction unit 112 identifies the type of fingerprint image based on the input information and can perform the above processing using the threshold corresponding to the identified type. Alternatively, the feature point information extraction unit 112 may normalize the values ​​of the first estimated heatmap estimated from each fingerprint image and then compare them with the thresholds described above. In this case, the process of changing the threshold for each type of fingerprint image is unnecessary.

[0050] There are various methods for determining the coordinates that take place at a local maximum value. For example, the feature point information extraction unit 112 may determine that the pixel to be judged is the coordinate that takes place at a local maximum value if the value of that pixel is at its maximum value within a predetermined area (e.g., a 5x5 pixel area) centered on the pixel to be judged.

[0051] The feature point information extraction unit 112 can perform the above processing on each of the multiple first estimated heatmaps corresponding to each of the multiple directions. As a result, the coordinates where fingerprint feature points exist can be estimated from each of the multiple first estimated heatmaps corresponding to each of the multiple directions. The feature point information extraction unit 112 can estimate the union of the coordinates where fingerprint feature points exist, estimated from each of the multiple first estimated heatmaps corresponding to each of the multiple directions, as the coordinates where fingerprint feature points exist. In other words, by the above processing, the feature point information extraction unit 112 can estimate the "coordinates where fingerprint feature points exist" estimated from at least one first estimated heatmap as the coordinates where fingerprint feature points exist.

[0052] Another method for determining the coordinates that take the local maximum value is described below. The feature point information extraction unit 112 determines a predetermined area (e.g., a 5x5 pixel area) centered on the pixel to be judged, and then determines whether the value of the pixel to be judged is the maximum value in the three-dimensional space that includes the values ​​of the predetermined area in each of the multiple first estimated heatmaps. If the value of the pixel to be judged is the maximum value in the three-dimensional space, the feature point information extraction unit 112 can determine that the pixel to be judged is the coordinate that takes the local maximum value. In this example, since the coordinates that take the local maximum value are determined by considering multiple first estimated heatmaps, the calculation of the union described above becomes unnecessary. In this method, the inconvenience of repeatedly determining the same coordinate as the coordinate that takes the local maximum value from among multiple first estimated heatmaps can be suppressed.

[0053] Furthermore, if the number of coordinates where fingerprint feature points are estimated by the above process is enormous, the processing load on the computer will increase. Also, if there is a certain amount of information (position and direction components) on fingerprint feature points, fingerprint matching can be performed with a sufficiently high degree of accuracy. Therefore, the feature point information extraction unit 112 may estimate a predetermined number of coordinates from among the coordinates where fingerprint feature points are estimated by the above process, starting with those with larger values ​​in the first estimated heatmap, as coordinates where fingerprint feature points are present. By adopting this configuration, it is possible to suppress the inconvenience of generating feature point information that unnecessarily includes information on a large number of fingerprint feature points, thereby unnecessarily increasing the computer load.

[0054] Next, we will explain the process of estimating the direction of the fingerprint lines extending from the fingerprint feature points at each coordinate, for example, for each coordinate where a fingerprint feature point estimated as described above exists.

[0055] In one example, the feature point information extraction unit 112 compares the values ​​of multiple first estimated heatmaps corresponding to multiple directions for each coordinate where a fingerprint feature point is estimated to exist. The feature point information extraction unit 112 can then estimate the direction corresponding to the first estimated heatmap with the maximum value as the direction of the fingerprint line extending from the fingerprint feature point at each coordinate.

[0056] This example is effective when the numerical intervals for multiple predetermined directions are small and a large number of first estimated heatmaps are estimated. However, when the numerical intervals for multiple predetermined directions are large and a small number of first estimated heatmaps are estimated, the error in the estimation result (direction of fingerprint lines) may become large in this example. Therefore, the feature point information extraction unit 112 may be configured to perform the processing of the following other examples.

[0057] In another example, the feature point information extraction unit 112 compares the values ​​of multiple first estimated heatmaps corresponding to each of the multiple directions for each coordinate where a fingerprint feature point is estimated to exist. The feature point information extraction unit 112 then determines the direction corresponding to the first estimated heatmap with the maximum value, and the two directions adjacent to that direction. For example, if the multiple directions are defined in 60° increments, such as 0°, 60°, 120°, and 180°, and the direction corresponding to the first estimated heatmap with the maximum value is 120°, then the two directions adjacent to that direction are 60° and 180°.

[0058] The feature point information extraction unit 112 then uses data interpolation (e.g., quadratic interpolation) to determine the heat map values ​​in the directions between the three directions obtained, based on the heat map values ​​in each of the three directions obtained. For example, if the three directions obtained are 60°, 120°, and 180°, the feature point information extraction unit 112 uses data interpolation to determine the heat map values ​​in the direction between 60° and 120°. The feature point information extraction unit 112 also uses data interpolation to determine the heat map values ​​in the direction between 120° and 180°.

[0059] The feature point information extraction unit 112 then determines the direction that has the maximum value among the heatmap values ​​in each of the three directions obtained and the heatmap values ​​in the directions between those directions obtained by data interpolation processing. The feature point information extraction unit 112 can estimate the direction obtained in this way as the direction of the fingerprint line extending from the fingerprint feature point present at each coordinate.

[0060] Next, an example of the processing flow performed by the estimation device 10 will be explained using the flowchart in Figure 2. Note that the purpose here is to explain the processing flow. Details of each process have been described above, so explanations will be omitted here as appropriate.

[0061] In S10, the estimation device 10 estimates a first estimation heatmap from the fingerprint image. The first estimation heatmap represents the likelihood of the location of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points. That is, the first estimation heatmap represents the likelihood that fingerprint feature points exist at each location in the fingerprint image, extending in each direction.

[0062] In S11, the estimation device 10 extracts feature point information from the first estimation heatmap that indicates the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points.

[0063] <Modification> Now, a modification of this embodiment will be described. In this modification, the estimated heatmap generation unit 111 estimates one estimated heatmap representing the likelihood of the position of the fingerprint feature points from the fingerprint image. That is, the estimated heatmap estimated by the estimated heatmap generation unit 111 in this modification does not represent the likelihood of the direction of the fingerprint lines extending from the fingerprint feature points. In this modification, the estimated heatmap generation unit 111 is configured as a learning model that takes a fingerprint image as input and outputs the above-mentioned one estimated heatmap.

[0064] In this modified version, the feature point information extraction unit 112 can estimate the location of fingerprint feature points based on such a single estimated heatmap. The method is the same as the "process for estimating the coordinates where fingerprint feature points exist" described above. Alternatively, the feature point information extraction unit 112 can analyze the fingerprint image using other widely known techniques without using the estimated heatmap to estimate the direction of fingerprint lines extending from the fingerprint feature points.

[0065] As explained above, in this modified method, the location of fingerprint feature points is estimated using an estimated heatmap, and the direction of fingerprint lines extending from the fingerprint feature points is estimated by analyzing the fingerprint image using widely known techniques.

[0066] <Effects and Effects> The estimation device 10 of the second embodiment can achieve the same effects and effects as the estimation device 10 of the first embodiment.

[0067] Furthermore, the estimation device 10 can extract feature point information indicating the position of fingerprint feature points and the direction of fingerprint lines extending from them from a plurality of first estimation heatmaps corresponding to each of a plurality of predefined directions. The first estimation heatmap corresponding to the first direction among the plurality of directions represents the likelihood that a fingerprint feature point exists at each coordinate that is the starting point of a fingerprint line extending in the first direction.

[0068] With such an estimation device 10, the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points can be estimated with high accuracy. As a result, even with fingerprint images of poor quality that are difficult to match, it is possible to estimate the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points with high accuracy and perform fingerprint matching with sufficient accuracy.

[0069] <<Third Embodiment>> The estimation device 10 of the third embodiment further estimates a second estimation heatmap representing the likelihood of the feature point type of the fingerprint feature point, and uses the second estimation heatmap to estimate the feature point type of the fingerprint feature point detected from the fingerprint image. This will be explained in detail below.

[0070] The estimated heatmap generation unit 111 estimates a second estimated heatmap from the fingerprint image. That is, the estimated heatmap generation unit 111 estimates a first estimated heatmap and a second estimated heatmap from the fingerprint image. The method for estimating the first estimated heatmap is as described in the second embodiment.

[0071] The "second estimated heatmap" represents the likelihood of each feature point type for a fingerprint feature point. That is, the second estimated heatmap represents the likelihood that a fingerprint feature point of each feature point type exists at each location in the fingerprint image. Each feature point type includes at least one of the following: delta, core, endpoint, and bifurcation.

[0072] The estimated heatmap generation unit 111 can estimate at least one second estimated heatmap corresponding to each of the at least one predefined feature point type. The second estimated heatmap corresponding to the first feature point type among the at least one feature point type represents the likelihood that a fingerprint feature point of the first feature point type exists at each coordinate in the fingerprint image.

[0073] The estimated heatmap generation unit 111 is composed of, for example, a trained neural network. The configuration of such an estimated heatmap generation unit 111 is constructed using widely known techniques. The trained model takes a fingerprint image as input and outputs at least one second estimated heatmap corresponding to at least one feature point type. The following embodiment describes a characteristic training method of the trained model.

[0074] The feature point information extraction unit 112 extracts feature point information from the first estimated heatmap and the second estimated heatmap, indicating the position of the fingerprint feature points, the direction of the fingerprint lines extending from the fingerprint feature points, and the type of feature point of the fingerprint feature points. The process of extracting feature point information indicating the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points from the first estimated heatmap is as described in the second embodiment.

[0075] Here, we will explain an example of a process for extracting feature point information indicating the feature point type of fingerprint feature points from the second estimated heatmap.

[0076] The feature point information extraction unit 112 estimates the coordinates where fingerprint feature points exist based on the first estimated heatmap, and then determines whether the value of the second estimated heatmap at those coordinates exceeds a predetermined threshold. If a second estimated heatmap corresponding to each of multiple feature point types has been estimated, the feature point information extraction unit 112 determines whether the value of each of the multiple second estimated heatmaps at each coordinate where fingerprint feature points are estimated to exist exceeds the threshold. Through this determination, the feature point information extraction unit 112 can identify the second estimated heatmap whose value exceeds the threshold for each coordinate where fingerprint feature points are estimated to exist. It should be noted that there may be coordinates where fingerprint feature points are estimated to exist where none of the values ​​of the second estimated heatmaps exceed the threshold.

[0077] If a second estimated heatmap exists at a coordinate where a fingerprint feature point is estimated to exist, and the value exceeds a threshold, the feature point information extraction unit 112 estimates "the feature point type corresponding to that second estimated heatmap" as the feature point type of the fingerprint feature point present at that coordinate. For example, if the value of the second estimated heatmap corresponding to the first feature point type exceeds a threshold at a first coordinate where a fingerprint feature point is estimated to exist, the feature point information extraction unit 112 estimates the fingerprint feature point present at the first coordinate as the first feature point type.

[0078] Next, an example of the processing flow performed by the estimation device 10 will be explained using the flowchart in Figure 6. Note that the purpose here is to explain the processing flow. Details of each process have been described above, so explanations will be omitted here as appropriate.

[0079] In S20, the estimation device 10 estimates a first estimation heatmap and a second estimation heatmap from the fingerprint image. The first estimation heatmap represents the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points. That is, the first estimation heatmap represents the likelihood that fingerprint feature points exist at each position in the fingerprint image, extending in each direction. The second estimation heatmap represents the likelihood of the feature point type of the fingerprint feature points. That is, the second estimation heatmap represents the likelihood that fingerprint feature points of each feature point type exist at each position in the fingerprint image.

[0080] In S21, the estimation device 10 extracts feature point information from the first estimation heatmap and the second estimation heatmap, indicating the position of the fingerprint feature points, the direction of the fingerprint lines extending from the fingerprint feature points, and the type of feature point of the fingerprint feature points.

[0081] Other components of the estimation device 10 can be the same as those in the first and second embodiments.

[0082] The estimation device 10 of the third embodiment can achieve the same effects as the estimation devices 10 of the first and second embodiments. Furthermore, the estimation device 10 of the third embodiment can extract feature point information indicating the feature point type of a fingerprint feature point from at least one second estimation heatmap corresponding to each of the at least one predefined feature point type. The second estimation heatmap corresponding to the first feature point type among the at least one feature point type represents the likelihood that a fingerprint feature point of the first feature point type exists at each coordinate in the fingerprint image.

[0083] With such an estimation device 10, the position of fingerprint feature points, the direction of fingerprint lines extending from the fingerprint feature points, and the type of feature points of the fingerprint feature points can be estimated with high accuracy. As a result, even with fingerprint images of poor quality that are difficult to match, it is possible to estimate the position of fingerprint feature points, the direction of fingerprint lines extending from the fingerprint feature points, and the type of feature points of the fingerprint feature points with high accuracy, and perform fingerprint matching with sufficient accuracy.

[0084] <<Fourth Embodiment>> The learning device 20 of the fourth embodiment has the function of learning the estimation device 10 using correct fingerprint feature point information. This will be described in detail below.

[0085] <Hardware Configuration> First, an example of the hardware configuration of the learning device 20 will be described. Each functional part of the learning device 20 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various modifications to the implementation method and the device. The software includes programs that are stored in the device from the time of shipment, as well as programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0086] Figure 4 is a block diagram illustrating the hardware configuration of the learning device 20. The explanation of each element in the hardware configuration of Figure 4 was provided in the second embodiment, so the explanation is omitted here.

[0087] <Functional Configuration> Figure 7 shows an example of a functional block diagram of the learning device 20. The learning device 20 includes an estimated heatmap generation unit 111, a learning unit 21, and a correct answer heatmap calculation unit 23. The learning unit 21 includes a first learning unit 211.

[0088] First, the training data used in this embodiment will be described. In this embodiment, training data consisting of pairs of fingerprint images and ground truth fingerprint feature point information is used.

[0089] The "ground truth fingerprint feature point information" indicates the correct position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points in the paired fingerprint images. The ground truth fingerprint feature point information may also indicate the feature point type of the fingerprint feature points in the paired fingerprint images. The training data may be generated by the operator verifying the fingerprint images and inputting the ground truth, or by other means.

[0090] Next, we will explain the details of each functional part of the learning device 20.

[0091] The estimated heatmap generation unit 111 estimates a first estimated heatmap from the fingerprint images that serve as training data. The other configurations of the estimated heatmap generation unit 111 are the same as those of the first to third embodiments.

[0092] The ground truth heatmap calculation unit 23 generates a first ground truth heatmap from the ground truth fingerprint feature point information, representing the likelihood of the location of the fingerprint feature point and the direction of the fingerprint lines extending from the fingerprint feature point. The ground truth fingerprint feature point information pinpoints and clearly indicates the location within the fingerprint image where a fingerprint feature point extending in a predetermined direction exists. If the ground truth in the training data is represented by such a single point within a narrow range, it may become sensitive to errors, for example, and training may become unstable. Therefore, in this embodiment, a first ground truth heatmap is generated from the ground truth fingerprint feature point information, representing the likelihood that a fingerprint feature point extending in each direction exists at each location within the fingerprint image, and this is used for training.

[0093] The "first correct answer heatmap" represents the likelihood that fingerprint feature points extending in each direction exist at each location within the fingerprint image.

[0094] The ground truth heatmap calculation unit 23 can calculate (generate) multiple first ground truth heatmaps corresponding to each of a predetermined number of directions. The first ground truth heatmap corresponding to the first direction among the multiple directions represents the likelihood that a fingerprint feature point exists at each coordinate in the fingerprint image that is the starting point of a fingerprint line extending in the first direction.

[0095] The predefined multiple directions are as described in the second embodiment. The ground truth heatmap calculation unit 23 can generate a number of first ground truth heatmaps corresponding to the number of first estimated heatmaps output by the estimated heatmap generation unit 111 (i.e., the same number). The ground truth heatmap calculation unit 23 can generate multiple first ground truth heatmaps corresponding to the same directions as the multiple first estimated heatmaps output by the estimated heatmap generation unit 111.

[0096] Here, we will explain the process for generating the first correct answer heatmap. The correct answer heatmap calculation unit 23 can calculate the first correct answer heatmap by distributing the probability of correct answers to the periphery using a widely known distribution such as a normal distribution. Below, we will explain an example of the process for generating the first correct answer heatmap.

[0097] In one example, as shown in Figure 8, the ground truth heatmap calculation unit 23 generates a positional ground truth heatmap and a plurality of directional ground truth heatmaps corresponding to each of a plurality of predefined directions. Then, based on the positional ground truth heatmap and each of the plurality of directional ground truth heatmaps, the ground truth heatmap calculation unit 23 generates a plurality of first ground truth heatmaps corresponding to each of the plurality of directions. Based on the positional ground truth heatmap and the directional ground truth heatmaps corresponding to the first direction, the ground truth heatmap calculation unit 23 can generate a first ground truth heatmap corresponding to the first direction.

[0098] The "position-correct heatmap" represents the likelihood of a fingerprint feature point existing at each location in the fingerprint image. The position-correct heatmap calculation unit 23 can generate a position-correct heatmap that plots a distribution spreading outwards from the location (correct) of the fingerprint feature point indicated by the correct fingerprint feature point information. The position-correct heatmap calculation unit 23 can use a distribution such as that shown in Figure 9, for example. This distribution shows the probability of a difference from a reference value occurring. This distribution may be a normal distribution or any other distribution.

[0099] The ground truth heatmap calculation unit 23 uses a predefined distribution to determine the probability of occurrence for each position in the fingerprint image, corresponding to the distance (difference) from the position (ground truth) of the fingerprint feature point indicated in the ground truth fingerprint feature point information. Then, based on the probability of occurrence determined for each position in the fingerprint image, the ground truth heatmap calculation unit 23 determines the likelihood of the existence of a fingerprint feature point at each position in the fingerprint image (the value of each coordinate in the positional ground truth heatmap). The smaller the distance (difference) from the position (ground truth) of the fingerprint feature point indicated in the ground truth fingerprint feature point information, the higher the likelihood of the fingerprint feature point existing at that position. Naturally, the likelihood of the position (ground truth) of the fingerprint feature point indicated in the ground truth fingerprint feature point information is the highest. For example, the ground truth heatmap calculation unit 23 may use the probability of occurrence determined for each position in the fingerprint image as the likelihood of the existence of a fingerprint feature point at that position in the fingerprint image. Alternatively, the correct answer heatmap calculation unit 23 may use the value obtained by normalizing the probability of occurrence corresponding to each position in the fingerprint image according to a predetermined rule as the likelihood of the existence of a fingerprint feature point at each position in the fingerprint image.

[0100] The "direction-corrected heatmap" shows the likelihood that the fingerprint lines extending from each fingerprint feature point at each coordinate are pointing in each of a predefined number of directions. As shown in Figure 8, multiple direction-corrected heatmaps are generated, corresponding to each of the multiple directions. The direction-corrected heatmap corresponding to the first of the multiple directions shows the likelihood that the fingerprint lines extending from each fingerprint feature point at each coordinate are pointing in the first direction.

[0101] The correct heatmap calculation unit 23 identifies the likelihood that a fingerprint line extending from each fingerprint feature point indicated by correct fingerprint feature point information faces each of a plurality of directions, based on a distribution indicating the occurrence probability of a difference from the direction (correct answer) of the fingerprint line extending from the fingerprint feature point indicated by the correct fingerprint feature point information. The distribution may be a normal distribution or any other distribution.

[0102] Here, this process will be described using a specific example. In this example, it is assumed that a plurality of directions are defined in 60° increments such as 0°, 60°, 120°, and 180°. Then, in the correct fingerprint feature point information, (x １ , y １ ) coordinates and (x ２ , y ２ ) it is assumed that it is indicated that a fingerprint feature point exists at the coordinates. Further, (x １ , y １ ) the direction of the fingerprint line extending from the coordinates is 70°, and (x ２ , y ２ ) it is assumed that it is indicated that the direction of the fingerprint line extending from the coordinates is 5°.

[0103] Here, the process of generating a direction correct heatmap corresponding to 60° will be described. First, (x １ , y １ ) focus on the fingerprint feature point existing at the coordinates. The direction (correct answer) of the fingerprint line extending from this fingerprint feature point indicated by the correct fingerprint feature point information is 70°. Therefore, the correct heatmap calculation unit 23 obtains 10°, which is the difference between 70° (correct answer) and 60° (the corresponding direction). Next, the correct heatmap calculation unit 23 refers to the distribution as shown in FIG. 9 to obtain the occurrence probability of the difference of 10°. Then, the correct heatmap calculation unit 23, based on the obtained occurrence probability, obtains (x １ , y １ ) determines the coordinate value (likelihood). For example, the correct heatmap calculation unit 23 (x １ , y １ ) uses the occurrence probability obtained corresponding to the coordinates as it is for (x １ , y １The likelihood that the direction of the fingerprint line extending from the fingerprint feature point located at the coordinates of (x) is 60° may also be used. In addition, the ground truth heatmap calculation unit 23 calculates (x １ , y １ The probability of occurrence calculated in relation to the coordinates of (x) is normalized according to a predetermined rule, and the value after this normalization is (x １ , y １ This can also be defined as the likelihood that the direction of the fingerprint line extending from a fingerprint feature point located at the coordinates of ) is 60°.

[0104] Next, (x ２ , y ２ The unit focuses on the fingerprint feature point located at the coordinates (x). The direction of the fingerprint line extending from this fingerprint feature point, as shown in the ground truth fingerprint feature point information, is 5°. Therefore, the ground truth heatmap calculation unit 23 calculates 55°, which is the difference between 5° (ground truth) and 60° (corresponding direction). Next, the ground truth heatmap calculation unit 23 calculates the probability of occurrence of the difference 55° by referring to the distribution shown in Figure 9. Then, based on the calculated probability of occurrence, the ground truth heatmap calculation unit 23 calculates the (x) in the direction ground truth heatmap corresponding to 60°. ２ , y ２ The coordinate values ​​(likelihood) of (x) are determined. For example, the correct heatmap calculation unit 23 determines the coordinate values ​​(likelihood) of (x ２ , y ２ The probability of occurrence calculated corresponding to the coordinates of (x) is used as is. ２ , y ２ The likelihood that the direction of the fingerprint line extending from the fingerprint feature point located at the coordinates of (x) is 60° may also be used. In addition, the ground truth heatmap calculation unit 23 calculates (x ２ , y ２ The probability of occurrence calculated in relation to the coordinates of (x) is normalized according to a predetermined rule, and the value after this normalization is (x ２ , y ２ This can also be defined as the likelihood that the direction of the fingerprint line extending from a fingerprint feature point located at the coordinates of ) is 60°.

[0105] Here, it is assumed that the ground truth fingerprint feature point information indicates the presence of fingerprint feature points at two coordinates. However, in reality, the ground truth fingerprint feature point information indicates the presence of fingerprint feature points at many more coordinates. The ground truth heatmap calculation unit 23 performs the above processing for each coordinate where the ground truth fingerprint feature point information indicates the presence of fingerprint feature points, and can obtain the value in the directional ground truth heatmap corresponding to 60° for each coordinate. Then, based on the values ​​of each coordinate obtained in this way, the ground truth heatmap calculation unit 23 can generate the directional ground truth heatmap corresponding to 60°.

[0106] The correct heatmap calculation unit 23 can perform the above processing for each of the predefined directions and generate multiple correct heatmaps for each of the directions.

[0107] The ground truth heatmap calculation unit 23 generates multiple first ground truth heatmaps corresponding to each of the multiple directions, based on the positional ground truth heatmap generated as described above and each of the multiple directional ground truth heatmaps.

[0108] Here, we will explain the process of generating a first ground truth heatmap corresponding to the first direction, based on a position-correct heatmap and a direction-correct heatmap corresponding to the first direction.

[0109] The correct heatmap calculation unit 23 calculates a value for each coordinate using a predetermined calculation formula that utilizes the value of the positional correct heatmap and the value of the directional correct heatmap corresponding to the first direction. The correct heatmap calculation unit 23 then uses the value of each coordinate in the first correct heatmap corresponding to the first direction as the calculated value of each coordinate obtained by the above calculation formula. An example of the calculation formula is, but is not limited to, the product of the value of the positional correct heatmap and the value of the directional correct heatmap corresponding to the first direction as the calculated value.

[0110] The process will be explained using Figure 8. In Figure 8, the coordinate value of the top left is a in the position-corrected heatmap. １ And the value of the same coordinate in the direction-correct heatmap corresponding to the 0° direction is b. １In this case, the correct heatmap calculation unit 23 calculates the value of the same coordinate in the first correct heatmap corresponding to the 0-degree direction as a １ and b １ Let it be the product of the two.

[0111] Furthermore, in Figure 8, the value at the same coordinate in the direction-correct heatmap corresponding to the 60° direction is b'. １ In this case, the correct heatmap calculation unit 23 calculates the value of the same coordinate in the first correct heatmap corresponding to the 60-degree direction as a １ and b' １ Let it be the product of the two.

[0112] In this way, the ground truth heatmap calculation unit 23 can calculate the product of the value (likelihood) shown in the positional ground truth heatmap and the value (likelihood) shown in the directional ground truth heatmap for each coordinate in each of the multiple directions. Then, the ground truth heatmap calculation unit 23 can generate a first ground truth heatmap for each direction that shows this product for each coordinate.

[0113] Returning to Figure 7, the first learning unit 211 updates the parameters of the model that estimates the first estimated heatmap based on the first estimated heatmap and the first ground truth heatmap. The first learning unit 211 can update the parameters of the model that estimates the first estimated heatmap by using the difference between the first estimated heatmap and the first ground truth heatmap as a loss. Widely known techniques such as MSE Loss, L1 Loss, and PSNR can be used to calculate the loss. The first learning unit 211 can update the parameters of the model that estimates the first estimated heatmap so that the loss is minimized.

[0114] Next, an example of the processing flow performed by the learning device 20 will be explained using the flowchart in Figure 10. Note that the purpose here is to explain the processing flow. Details of each process have been described above, so explanations will be omitted here as appropriate.

[0115] In S30, the estimation device 10 estimates a first estimated heatmap from the fingerprint images of the training data, which represents the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points.

[0116] In S31, the estimation device 10 generates a first ground truth heatmap from the ground truth fingerprint feature point information of the training data, representing the likelihood of the position of the fingerprint feature point and the direction of the fingerprint line extending from the fingerprint feature point.

[0117] In S32, the estimation device 10 updates the parameters of the model that estimates the first estimated heatmap based on the first estimated heatmap and the first ground truth heatmap.

[0118] Note that the processing order of S30 and S31 is not limited to this example. S30 may be performed after S31, or S30 and S31 may be performed in parallel.

[0119] <Modified Versions> Now, modified versions of this embodiment will be described.

[0120] "Variation 1" Hyperparameter σ that controls the spread of the normal distribution used in the process of generating a position-correct heatmap １ And, the hyperparameter σ controls the spread of the normal distribution used in the process of generating a directionally correct heatmap. ２ These can be the same value or different values.

[0121] However, the range of differences considered in the process of generating a position-correct heatmap is the range of the image size, whereas the range of differences considered in the process of generating a direction-correct heatmap is 0° to 360°. Thus, the range of differences considered differs for each. Therefore, taking this difference into account, σ １ and σ ２ It is preferable to set the value to an appropriate value according to each range. That is, σ １ and σ ２ It is preferable to set different values ​​for σ. For example, taking into account the differences in the above ranges, １ σ ２ It can be made smaller than this. １ and σ ２ By appropriately setting the value of , the estimation accuracy of the first estimated heatmap is improved.

[0122] "Variation 2" σ １ and σ ２The value of may be determined by the following method. For example, feature point information is extracted from the first ground truth heatmap obtained in the above process using the feature point information extraction unit 112. Next, the extracted fingerprint feature point information is compared with the ground truth fingerprint feature point information. Then, the value for which this comparison result matches (for example, the relatively large value among those that satisfy the condition, for example, the maximum value) is determined as σ １ and σ ２ This can be determined as the value of σ. In addition, the value for which the above comparison result matches in a predetermined proportion or more of the fingerprint feature points shown in the correct fingerprint feature point information (for example, the relatively large value among those that satisfy the condition, for example, the maximum value) is σ １ and σ ２ It may be determined as a value.

[0123] σ １ and σ ２ The smaller the value of σ, the smaller the spread of the distribution, making it possible to estimate information about nearby feature points, but the learning becomes more difficult to converge. １ and σ ２ Increasing the value within an appropriate range makes it easier to learn.

[0124] "Variation 3" σ １ and σ ２ The value of σ may be smaller than the average distance between fingerprint feature points. For example, σ may be less than 3. The positions between fingerprint feature points may be subject to a superimposed normal distribution, potentially resulting in high ground truth heatmap values ​​even in areas where no feature points exist.

[0125] "Variation 4" σ １ and σ ２ The value of may be changed according to the learning stage. For example, in the early stages of learning, σ １ and σ ２ The value of is made relatively large, and once learning has progressed to a certain extent, σ １ and σ ２ The value of σ can be made relatively small. １ and σ ２ The smaller the value of σ, the more accurate the estimation of feature point information becomes, but the more difficult the learning becomes. For this reason, in the early stages of learning, １ and σ ２The value of is made relatively large to make learning easier, and once learning has progressed to a certain extent, σ １ and σ ２ You can improve the estimation accuracy by making the value of relatively smaller.

[0126] σ １ and σ ２ There are various timings for switching the value. For example, it could be when the loss falls below a predetermined value, when the model parameters have been updated a predetermined number of times, or for other reasons.

[0127] "Modification 5" The normal distribution used in the process of generating a location-correct heatmap can be expressed, for example, by the following equation (1).

[0128]

[0129] The geometrically accurate heatmap generated using the normal distribution described above has gently curved peaks, but sharpening the peaks improves the accuracy of feature point estimation. Therefore, the distribution represented by equation (2) below may be used in the process of generating geometrically accurate heatmaps. The distribution represented by equation (2) below has sharp peaks because the L2 norm is replaced with the L1 norm.

[0130]

[0131] In addition, the distribution represented by equation (3) below may be used in the process of generating a position-correct heatmap. The distribution represented by equation (3) below not only sharpens the peaks but also eliminates the need for exponential calculations using exp(), resulting in faster heatmap creation.

[0132]

[0133] Furthermore, in the normal distribution used in the process of generating a directionally correct heatmap, the part of equations (1) to (3) above that was used to calculate the coordinate distance (L2 norm, L1 norm) may be calculated as a difference in direction.

[0134] "Modification 6" The estimated heatmap generation unit 111 may further estimate a second estimated heatmap from the fingerprint images which are used as training data.

[0135] The ground truth heatmap calculation unit 23 may generate a second ground truth heatmap representing the likelihood of the feature point type of the fingerprint feature point from the ground truth fingerprint feature point information. The ground truth fingerprint feature point information clearly indicates the location within the fingerprint image where a fingerprint feature point of a predetermined feature point type exists. If the ground truth in the training data is represented by such a single point within a narrow range, it may become sensitive to errors, for example, and training may become unstable. Therefore, in this modified example, a second ground truth heatmap representing the likelihood of the existence of a fingerprint feature point of each feature point type at each location within the fingerprint image is generated from the ground truth fingerprint feature point information and used for training.

[0136] The "Second Correct Heatmap" represents the likelihood that a fingerprint feature point of each feature point type exists at each location within the fingerprint image.

[0137] The ground truth heatmap calculation unit 23 can calculate (generate) at least one second ground truth heatmap corresponding to each of the at least one predefined feature point type. The second ground truth heatmap corresponding to the first feature point type among the at least one feature point type represents the likelihood that a fingerprint feature point of the first feature point type exists at each coordinate in the fingerprint image.

[0138] When the ground truth heatmap calculation unit 23 generates a second ground truth heatmap corresponding to the first feature point type, it can generate a second ground truth heatmap that plots a distribution spreading out from the position (ground truth) of the fingerprint feature point of the first feature point type indicated by the ground truth fingerprint feature point information. The ground truth heatmap calculation unit 23 can use a distribution such as that shown in Figure 9, for example. This distribution shows the probability of a difference from a reference value occurring. This distribution may be a normal distribution or any other distribution.

[0139] The ground truth heatmap calculation unit 23 uses a predefined distribution to determine the probability of occurrence for each position in the fingerprint image, corresponding to the distance (difference) from the position (ground truth) of the fingerprint feature point of the first feature point type indicated in the ground truth fingerprint feature point information. Then, based on this probability of occurrence, the ground truth heatmap calculation unit 23 determines the likelihood of the existence of a fingerprint feature point of the first feature point type at each position in the fingerprint image. The smaller the distance (difference) from the position (ground truth) of the fingerprint feature point of the first feature point type indicated in the ground truth fingerprint feature point information, the higher the likelihood of the existence of the fingerprint feature point of the first feature point type at that position. Naturally, the likelihood of the position (ground truth) of the fingerprint feature point of the first feature point type indicated in the ground truth fingerprint feature point information is the highest. For example, the ground truth heatmap calculation unit 23 may use the probability of occurrence determined for each position in the fingerprint image as the likelihood of the existence of a fingerprint feature point of the first feature point type at each position in the fingerprint image. Alternatively, the correct answer heatmap calculation unit 23 may use the value obtained by normalizing the probability of occurrence corresponding to each position in the fingerprint image according to a predetermined rule as the likelihood that a fingerprint feature point of the first feature point type exists at each position in the fingerprint image.

[0140] The first learning unit 211 updates the parameters of the model that estimates the second estimated heatmap based on the second estimated heatmap and the second ground truth heatmap. The first learning unit 211 can update the parameters of the model that estimates the second estimated heatmap by using the difference between the second estimated heatmap and the second ground truth heatmap as a loss. Widely known techniques such as MSE Loss, L1 Loss, and PSNR can be used to calculate the loss. The first learning unit 211 can update the parameters of the model that estimates the second estimated heatmap so that the loss is minimized.

[0141] "Modification 7" As described in the modification of the second embodiment, the estimated heatmap generation unit 111 may estimate one estimated heatmap representing the likelihood of the position of the fingerprint feature points from the fingerprint image. That is, the estimated heatmap estimated by the estimated heatmap generation unit 111 does not have to represent the likelihood of the direction of the fingerprint lines extending from the fingerprint feature points. In this modification, a learning model is trained that takes a fingerprint image as input and outputs the above-mentioned one estimated heatmap.

[0142] In this modified case, the ground truth heatmap calculation unit 23 generates a first ground truth heatmap that plots a distribution spreading around coordinates corresponding to the positions of fingerprint feature points indicated in the ground truth fingerprint feature point information. In other words, the ground truth heatmap calculation unit 23 can generate the above-mentioned positional ground truth heatmap as the first ground truth heatmap.

[0143] The first learning unit 211 can then update the parameters of the model that estimates the one estimated heatmap by using the difference between the one estimated heatmap and the first ground truth heatmap as a loss.

[0144] "Modification 8" The learning device 20 may, after estimating a first estimated heatmap from the fingerprint images which are used as learning data, extract feature point information from the first estimated heatmap. This process is carried out by the estimated heatmap generation unit 111 and the feature point information extraction unit 112.

[0145] The learning device 20 may update the parameters of the model that estimates the one estimated heatmap by using the difference between the feature point information extracted from the first estimated heatmap and the ground truth fingerprint feature point information as a loss. Widely known techniques such as MSE Loss, L1 Loss, and PSNR can be used to calculate the loss. In this example, after calculating the difference between the feature point information extracted from the first estimated heatmap and the ground truth fingerprint feature point information as a loss, the learning device 20 can calculate the gradient value of the parameters of the feature point information estimation unit 11 from this difference (loss). Then, the learning device 20 can calculate the parameters of the feature point information estimation unit 11 from this gradient value and update the parameters.

[0146] <Effects> According to the learning device 20 of the fourth embodiment, a first estimated heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points can be estimated from the fingerprint images of the training data. The learning device 20 can also generate a first ground truth heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from the ground truth fingerprint feature point information of the training data. The learning device 20 can then update the parameters of the model that estimates the first estimated heatmap by using the difference between the first estimated heatmap and the first ground truth heatmap as a loss. With such a learning device 20, the parameters of the model that estimates the first estimated heatmap can be appropriately updated.

[0147] Furthermore, the learning device 20 can generate a first ground truth heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points, using the characteristic processing described above. With such a learning device 20, as described above, it is possible to generate a first ground truth heatmap that is easy to learn and has appropriate content. As a result, appropriate learning is performed, and the parameters of the model that estimates the first estimated heatmap are appropriately updated.

[0148] <<Fifth Embodiment>> The learning device 20 of the fifth embodiment has a function to learn the estimation device 10 using a different method than that of the fourth embodiment. The learning device 20 of this embodiment learns the estimation device 10 without ground truth fingerprint feature point information. This will be explained in detail below.

[0149] <Hardware Configuration> An example of the hardware configuration of the learning device 20 in this embodiment is the same as in the fourth embodiment.

[0150] <Functional Configuration> Figure 11 shows an example of a functional block diagram of the learning device 20. The learning device 20 includes a feature point information estimation unit 11, a learning unit 21, and a matching unit 31. The feature point information estimation unit 11 includes an estimation heatmap generation unit 111 and a feature point information extraction unit 112. The learning unit 21 includes a second learning unit 212.

[0151] First, the training data used in the learning of this embodiment will be described. In this embodiment, multiple fingerprint images are used as training data. The training data is capable of identifying whether or not multiple fingerprint images record the same fingerprint. For example, each of the multiple fingerprint images may be assigned fingerprint identification information. Fingerprint identification information is information that identifies multiple fingerprints from each other. Multiple fingerprint images that record the same fingerprint are assigned the same fingerprint identification information.

[0152] In addition, multiple pairs of fingerprint images may be created in the training data beforehand. Each pair may then be assigned a correct label indicating whether or not it records the same fingerprint.

[0153] Next, we will explain the details of each functional part of the learning device 20.

[0154] The estimated heatmap generation unit 111 estimates a first estimated heatmap from the first fingerprint image, which is the training data. Then, the feature point information extraction unit 112 extracts feature point information from the first estimated heatmap estimated from the first fingerprint image.

[0155] The estimated heatmap generation unit 111 may further estimate a second estimated heatmap from the first fingerprint image, which is the training data. The feature point information extraction unit 112 may then extract feature point information from the first estimated heatmap and the second estimated heatmap estimated from the first fingerprint image.

[0156] Hereinafter, "feature point information extracted from the first estimated heatmap estimated from the first fingerprint image" or "feature point information extracted from the first estimated heatmap and the second estimated heatmap estimated from the first fingerprint image" will be referred to as "first estimated fingerprint feature point information."

[0157] Furthermore, the estimated heatmap generation unit 111 estimates a first estimated heatmap from the second fingerprint image, which is training data. Then, the feature point information extraction unit 112 extracts feature point information from the first estimated heatmap estimated from the second fingerprint image.

[0158] The estimated heatmap generation unit 111 may further estimate a second estimated heatmap from the second fingerprint image, which is training data. The feature point information extraction unit 112 may then extract feature point information from the first estimated heatmap and the second estimated heatmap estimated from the second fingerprint image.

[0159] Hereinafter, "feature point information extracted from the first estimated heatmap estimated from the second fingerprint image" or "feature point information extracted from the first estimated heatmap and the second estimated heatmap estimated from the second fingerprint image" will be referred to as "second estimated fingerprint feature point information."

[0160] The second fingerprint image is a different fingerprint image from the first fingerprint image. Note that the first and second fingerprint images may either record the same fingerprint or different fingerprints. As mentioned above, the training data already reveals whether the first and second fingerprint images record the same fingerprint or different fingerprints.

[0161] The other configurations of the estimated heatmap generation unit 111 and the feature point information extraction unit 112 are the same as in the first to third embodiments.

[0162] The matching unit 31 outputs a matching score between the first estimated fingerprint feature point information estimated from the first fingerprint image and the second estimated fingerprint feature point information estimated from the second fingerprint image.

[0163] The matching score is the degree of similarity between the first estimated fingerprint feature point information and the second estimated fingerprint feature point information. The calculation of similarity between the two sets of information can be achieved using widely known techniques.

[0164] For example, the matching unit 31 may be composed of a pre-trained neural network. The model is trained so that when the first fingerprint image and the second fingerprint image record the same fingerprint, the matching score between the first estimated fingerprint feature point information and the second estimated fingerprint feature point information is high. The model is also trained so that when the first fingerprint image and the second fingerprint image record different fingerprints, the matching score between the first estimated fingerprint feature point information and the second estimated fingerprint feature point information is low.

[0165] The second learning unit 212 updates the parameters of the model that estimates estimated fingerprint feature point information and the model that outputs the matching score, based on the matching score and the correct answer as to whether the first fingerprint image and the second fingerprint image are fingerprint images that record the same fingerprint.

[0166] The second learning unit 212 updates the parameters of the model that estimates estimated fingerprint feature point information and the model that outputs the matching score so that the matching score is higher when the first fingerprint image and the second fingerprint image are fingerprint images that record the same fingerprint. The second learning unit 212 also updates the parameters of the model that estimates estimated fingerprint feature point information and the model that outputs the matching score so that the matching score is lower when the first fingerprint image and the second fingerprint image are fingerprint images that record different fingerprints.

[0167] Next, an example of the processing flow performed by the learning device 20 will be explained using the flowchart in Figure 12. The purpose here is simply to explain the processing flow. Details of each process have been described above, so explanations will be omitted here as appropriate.

[0168] In S40, the learning device 20 estimates first estimated fingerprint feature point information from the first fingerprint image of the learning data.

[0169] In S41, the learning device 20 estimates second estimated fingerprint feature point information from the second fingerprint image of the learning data.

[0170] In S42, the learning device 20 calculates a matching score between the first estimated fingerprint feature point information and the second estimated fingerprint feature point information.

[0171] In S43, the learning device 20 updates the parameters of the model that estimates estimated fingerprint feature point information and the model that outputs the matching score, based on the matching score and the correct answer as to whether the first fingerprint image and the second fingerprint image are fingerprint images that record the same fingerprint.

[0172] <Modification> Now, a modification of this embodiment will be described. Figure 13 shows an example of a functional block diagram of the learning device 20 of this modified embodiment. It differs from the example in Figure 11 in that the matching unit 31 includes a normalization unit 311, a feature extraction unit 312, and a similarity calculation unit 313.

[0173] The normalization unit 311 generates a first normalized fingerprint image and a second normalized fingerprint image by performing at least one of orientation adjustment, position adjustment, and cropping on at least one of the first fingerprint image and the second fingerprint image.

[0174] The normalization unit 311 can perform at least one of the orientation adjustment, position adjustment, and cropping described above based on the "first estimated fingerprint feature point information" and "second estimated fingerprint feature point information" output from the feature point information estimation unit 11.

[0175] "Orientation adjustment" is a process that changes (rotates) at least one of the orientations of the fingerprint in the first fingerprint image and the second fingerprint image so that the orientation of the fingerprint in the first fingerprint image and the orientation of the fingerprint in the second fingerprint image are aligned (match).

[0176] "Position adjustment" is a process that translates at least one of the positions of the fingerprint in the first fingerprint image and the second fingerprint image so that the positions of the fingerprint in the first fingerprint image and the positions of the fingerprint in the second fingerprint image are aligned (match).

[0177] "Cropping" is the process of cutting out a portion of a fingerprint from at least one of the first fingerprint image and the second fingerprint image so that the portion of the fingerprint recorded in the first fingerprint image and the portion of the fingerprint recorded in the second fingerprint image are aligned (match).

[0178] This normalization process makes the conditions of the fingerprints recorded in the first fingerprint image and the fingerprints recorded in the second fingerprint image the same, making them suitable for matching.

[0179] The feature extraction unit 312 extracts features (first features) from the first normalized fingerprint image. The feature extraction unit 312 also extracts features (second features) from the second normalized fingerprint image. The feature extraction unit 312 can be configured, for example, with an existing neural network. The feature extraction unit 312 may extract a vector as a feature from one normalized fingerprint image, or it may extract a feature map or feature space as a feature. The feature extraction unit 312 may extract a one-dimensional feature vector, or it may extract a multi-dimensional feature vector of two or more dimensions.

[0180] The similarity calculation unit 313 calculates the similarity between the first feature and the second feature as the matching score. For example, the similarity calculation unit 313 may calculate the similarity between feature vectors using cosine similarity or Hamming distance. Alternatively, if the normalization unit 311 crops around the fingerprint feature points and rotates them so that the direction of the central feature points is unified, the feature may be calculated for each region and the cosine similarity may be calculated by brute force. The similarity calculation unit 313 may then use the average of a predetermined number of similarities, starting with the highest similarities, as the matching score. In addition, the similarity calculation unit 313 may use the average of all similarities calculated by brute force as the matching score. By cropping around the fingerprint feature points and unifying the direction of the feature points, matching can be performed for smaller regions, making it more resistant to missing or distorted fingerprints.

[0181] <<Example>> Here, we will describe an example using the estimation device 10.

[0182] <Example 1> In Example 1, a fingerprint image to be matched with the feature point information of multiple fingerprints registered in the fingerprint database is input to the estimation device 10. The estimation device 10 performs the processing described in the first to third embodiments and extracts feature point information from the fingerprint image to be matched.

[0183] The estimation device 10 then compares the extracted feature point information with the feature point information of multiple fingerprints registered in the fingerprint database. By using the estimation device 10, which can extract feature point information from fingerprint images with high accuracy, the accuracy of fingerprint matching is improved.

[0184] <Example 2> In Example 2, the fingerprint image to be registered in the fingerprint database is input to the estimation device 10. The estimation device 10 performs the processing described in the first to third embodiments and extracts feature point information from the fingerprint image to be registered.

[0185] The estimation device 10 then registers the extracted feature point information in the fingerprint database. By using the estimation device 10, highly accurate feature point information can be registered in the fingerprint database. As a result, the matching accuracy of matching using such a fingerprint database is improved.

[0186] Although this disclosure has been described above with reference to embodiments, this disclosure is not limited to the embodiments described above. Various modifications to the structure and details of this disclosure are possible, which can be understood by those skilled in the art within the scope of this disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0187] Furthermore, the flowchart used in the above explanation shows multiple steps (processes) in sequence. However, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content.

[0188] Some or all of the above embodiments may also be described as follows, but are not limited to the following: 1. An estimation device comprising: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the position of a fingerprint feature point and the direction of a fingerprint line extending from the fingerprint feature point from a fingerprint image; and a feature point information extraction means for extracting feature point information indicating the position of the fingerprint feature point and the direction of the fingerprint line from the first estimation heatmap. 2. The estimation device according to 1, wherein, when the number of first estimation heatmaps output by the estimation heatmap generation means is multiple, the feature point information extraction means extracts the feature point information indicating the position of the fingerprint feature point and the direction of the fingerprint line using the multiple first estimation heatmaps. 3. The estimation device according to 1, wherein the estimation heatmap generation means estimates a plurality of first estimation heatmaps corresponding to each of a plurality of predetermined directions, and the first estimation heatmap corresponding to a first direction among the plurality of directions represents the likelihood that the fingerprint feature point that is the starting point of the fingerprint line extending in the first direction exists at each coordinate. 4. The estimation apparatus according to 1, wherein the estimation heatmap generation means estimates a second estimation heatmap representing the likelihood of the feature point type of the fingerprint feature point from the fingerprint image, and the feature point information extraction means extracts the feature point information indicating the position of the fingerprint feature point, the direction of the fingerprint line, and the feature point type of the fingerprint feature point from the first estimation heatmap and the second estimation heatmap. 5. The estimation apparatus according to 4, wherein the estimation heatmap generation means estimates at least one second estimation heatmap corresponding to each of the at least one feature point type, and the second estimation heatmap corresponding to the first feature point type among the at least one feature point type represents the likelihood of the fingerprint feature point of the first feature point type existing at each coordinate. 6. The estimation apparatus according to 4, wherein the feature point type includes at least one of delta, core, endpoint, and branch point.7. An estimation method comprising: at least one computer estimating a first estimated heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points; and extracting feature point information indicating the positions of the fingerprint feature points and the directions of the fingerprint lines from the first estimated heatmap. 8. A recording medium storing a program that causes a computer to function as: an estimated heatmap generation means for estimating a first estimated heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points; and a feature point information extraction means for extracting feature point information indicating the positions of the fingerprint feature points and the directions of the fingerprint lines from the first estimated heatmap. 9. A learning device comprising: estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from a fingerprint image; ground truth heatmap calculation means for generating a first ground truth heatmap representing the likelihood of the position of the fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from ground truth fingerprint feature point information; and first learning means for updating the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and the first ground truth heatmap. 10. The learning device according to 9, wherein the first learning means updates the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and a loss calculated from the first ground truth heatmap. 11. The learning device according to 9, wherein the ground truth heatmap calculation means generates a number of first ground truth heatmaps corresponding to the number of first estimation heatmaps output by the estimation heatmap generation means. 12. The learning device according to 11, wherein the ground truth heatmap calculation means generates one first ground truth heatmap plotting a distribution that spreads around coordinates corresponding to the positions of the fingerprint feature points indicated in the ground truth fingerprint feature point information, when the number of first estimated heatmaps output by the estimated heatmap generation means is one.13. The learning device according to 11, wherein the ground truth heatmap calculation means, when the number of first estimated heatmaps output by the heatmap generation means is multiple, generates a positional ground truth heatmap showing the likelihood that the fingerprint feature point exists at each coordinate based on the ground truth fingerprint feature point information, generates multiple directional ground truth heatmaps showing the likelihood that the fingerprint lines extending from the fingerprint feature points at each coordinate are oriented in each of a predetermined number of directions based on the ground truth fingerprint feature point information, and generates a plurality of first ground truth heatmaps corresponding to each of the multiple directions using the positional ground truth heatmap and each of the plurality of directional ground truth heatmaps. 14. The learning device according to 13, wherein the ground truth heatmap calculation means identifies the likelihood that the fingerprint lines extending from each of the fingerprint feature points shown in the ground truth fingerprint feature point information are oriented in each of the multiple directions based on a distribution showing the probability of occurrence of a difference from the direction of the fingerprint lines extending from the fingerprint feature points shown in the ground truth fingerprint feature point information, and generates a plurality of directional ground truth heatmaps corresponding to each of the multiple directions based on the identified result. 15. The learning device according to 13, wherein the ground truth heatmap calculation means generates the positional ground truth heatmap plotting a distribution that spreads around the positions of the fingerprint feature points indicated by the ground truth fingerprint feature point information. 16. The learning device according to 13, wherein the ground truth heatmap calculation means calculates the product of the likelihood shown in the positional ground truth heatmap and the likelihood shown in the directional ground truth heatmap for each coordinate for each of the plurality of directions, and performs a process to generate the first ground truth heatmap showing the product of each coordinate. 17. A learning device comprising: a feature point information estimation means for estimating estimated fingerprint feature point information from a fingerprint image; a matching means for outputting a matching score between first estimated fingerprint feature point information estimated from a first fingerprint image and second estimated fingerprint feature point information estimated from a second fingerprint image different from the first fingerprint image; and a second learning means for updating the parameters of a model for estimating the estimated fingerprint feature point information and a model for outputting the matching score based on the matching score and the correct answer as to whether the first fingerprint image and the second fingerprint image are fingerprint images that record the same fingerprint.18. The learning device according to 17, comprising: a normalization means for generating a first normalized fingerprint image and a second normalized fingerprint image by performing at least one of orientation adjustment, position adjustment and cropping on at least one of the first fingerprint image and the second fingerprint image; a feature extraction means for extracting a first feature quantity and a second feature quantity from each of the first normalized fingerprint image and the second normalized fingerprint image; and a similarity calculation means for calculating the similarity between the first feature quantity and the second feature quantity as the matching score. 19. A learning method comprising: at least one computer estimating a first estimated heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points; generating a first ground truth heatmap from ground truth fingerprint feature point information that represents the likelihood of the positions of the fingerprint feature points and the directions of the fingerprint lines extending from the fingerprint feature points; and updating the parameters of a model that estimates the first estimated heatmap based on the first estimated heatmap and the first ground truth heatmap. 20. A recording medium storing a program that causes a computer to function as: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from a fingerprint image; a ground truth heatmap calculation means for generating a first ground truth heatmap representing the likelihood of the position of the fingerprint feature points and the direction of the fingerprint lines extending from the fingerprint feature points from ground truth fingerprint feature point information; and a first learning means for updating the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and the first ground truth heatmap.

[0189] 10 Estimation device 11 Feature point information estimation unit 111 Estimation heat map generation unit 112 Feature point information extraction unit 21 Learning unit 211 First learning unit 212 Second learning unit 23 Ground truth heat map calculation unit 31 Matching unit 311 Normalization unit 312 Feature quantity extraction unit 313 Similarity calculation unit 1A Processor 2A Memory 3A Input / Output I / F 4A Peripheral circuitry 5A Bus

Claims

1. An estimation device comprising: an estimation heatmap generation means for estimating a first estimation heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points; and a feature point information extraction means for extracting feature point information indicating the positions of the fingerprint feature points and the direction of the fingerprint lines from the first estimation heatmap.

2. The estimation apparatus according to claim 1, wherein, if the number of first estimated heatmaps output by the estimation heatmap generation means is multiple, the feature point information extraction means extracts the feature point information indicating the position of the fingerprint feature points and the direction of the fingerprint lines using the multiple first estimated heatmaps.

3. The estimation device according to claim 1, wherein the estimation heatmap generation means estimates a plurality of first estimation heatmaps corresponding to each of a plurality of predetermined directions, and the first estimation heatmap corresponding to the first direction among the plurality of directions represents the likelihood that the fingerprint feature point that is the starting point of the fingerprint line extending in the first direction exists at each coordinate.

4. The estimation device according to claim 1, wherein the estimation heatmap generation means estimates a second estimation heatmap representing the likelihood of the feature point type of the fingerprint feature point from the fingerprint image, and the feature point information extraction means extracts the feature point information indicating the position of the fingerprint feature point, the direction of the fingerprint line, and the feature point type of the fingerprint feature point from the first estimation heatmap and the second estimation heatmap.

5. The estimation device according to claim 4, wherein the estimation heatmap generation means estimates at least one second estimation heatmap corresponding to each of the at least one feature point type, and the second estimation heatmap corresponding to the first feature point type among the at least one feature point type represents the likelihood of the fingerprint feature point of the first feature point type existing at each coordinate.

6. The estimation device according to claim 4, wherein the feature point type includes at least one of delta, core, endpoint, and branch point.

7. An estimation method comprising: at least one computer estimating a first estimated heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points; and extracting feature point information indicating the positions of the fingerprint feature points and the direction of the fingerprint lines from the first estimated heatmap.

8. A recording medium that stores a program causing a computer to function as: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the positions of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from a fingerprint image; and a feature point information extraction means for extracting feature point information indicating the positions of the fingerprint feature points and the direction of the fingerprint lines from the first estimation heatmap.

9. A learning device comprising: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the position of fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from a fingerprint image; a ground truth heatmap calculation means for generating a first ground truth heatmap representing the likelihood of the position of the fingerprint feature points and the direction of fingerprint lines extending from the fingerprint feature points from ground truth fingerprint feature point information; and a first learning means for updating the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and the first ground truth heatmap.

10. The learning device according to claim 9, wherein the first learning means updates the parameters of the model that estimates the first estimated heatmap based on the first estimated heatmap and the loss calculated from the first ground truth heatmap.

11. The learning device according to claim 9, wherein the ground truth heatmap calculation means generates a number of first ground truth heatmaps corresponding to the number of first estimated heatmaps output by the estimated heatmap generation means.

12. The learning device according to claim 11, wherein the ground truth heatmap calculation means generates one first ground truth heatmap plotting a distribution that spreads around coordinates corresponding to the positions of the fingerprint feature points indicated in the ground truth fingerprint feature point information, when the number of first estimated heatmaps output by the estimated heatmap generation means is one.

13. The learning device according to claim 11, wherein, if the number of first estimated heatmaps output by the heatmap generation means is multiple, the ground truth heatmap calculation means generates a positional ground truth heatmap showing the likelihood that the fingerprint feature point exists at each coordinate based on the ground truth fingerprint feature point information, generates multiple directional ground truth heatmaps showing the likelihood that the fingerprint lines extending from the fingerprint feature point at each coordinate are oriented in each of a predetermined number of directions based on the ground truth fingerprint feature point information, and generates a plurality of first ground truth heatmaps corresponding to each of the multiple directions using the positional ground truth heatmap and each of the plurality of directional ground truth heatmaps.

14. The learning device according to claim 13, wherein the ground truth heatmap calculation means identifies the likelihood that the fingerprint lines extending from each of the fingerprint feature points indicated by the ground truth fingerprint feature point information are oriented in each of the plurality of directions, based on a distribution showing the probability of occurrence of a difference from the direction of the fingerprint lines extending from the fingerprint feature points indicated by the ground truth fingerprint feature point information, and generates a plurality of directional ground truth heatmaps corresponding to each of the plurality of directions based on the identified results.

15. The learning device according to claim 13, wherein the ground truth heatmap calculation means generates the positional ground truth heatmap plotting a distribution that spreads around the positions of the fingerprint feature points indicated by the ground truth fingerprint feature point information.

16. The learning device according to claim 13, wherein the ground truth heatmap calculation means calculates the product of the likelihood shown in the position ground truth heatmap and the likelihood shown in the direction ground truth heatmap for each coordinate for each of the plurality of directions, and performs the process of generating the first ground truth heatmap showing the product for each coordinate.

17. A learning device comprising: a feature point information estimation means for estimating estimated fingerprint feature point information from a fingerprint image; a matching means for outputting a matching score between first estimated fingerprint feature point information estimated from a first fingerprint image and second estimated fingerprint feature point information estimated from a second fingerprint image different from the first fingerprint image; and a second learning means for updating the parameters of a model for estimating the estimated fingerprint feature point information and the model for outputting the matching score, based on the matching score and the correct answer as to whether the first fingerprint image and the second fingerprint image are fingerprint images that record the same fingerprint.

18. The learning device according to claim 17, comprising: a normalization means for generating a first normalized fingerprint image and a second normalized fingerprint image by performing at least one of orientation adjustment, position adjustment and cropping on at least one of the first fingerprint image and the second fingerprint image; a feature extraction means for extracting a first feature quantity and a second feature quantity from each of the first normalized fingerprint image and the second normalized fingerprint image; and a similarity calculation means for calculating the similarity between the first feature quantity and the second feature quantity as the matching score.

19. A learning method comprising: at least one computer estimating a first estimated heatmap from a fingerprint image that represents the likelihood of the positions of fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points; generating a first ground truth heatmap from ground truth fingerprint feature point information that represents the likelihood of the positions of the fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points; and updating the parameters of a model that estimates the first estimated heatmap based on the first estimated heatmap and the first ground truth heatmap.

20. A recording medium storing a program that causes a computer to function as: an estimation heatmap generation means for estimating a first estimation heatmap representing the likelihood of the positions of fingerprint feature points and the directions of fingerprint lines extending from the fingerprint feature points from a fingerprint image; a ground truth heatmap calculation means for generating a first ground truth heatmap representing the likelihood of the positions of the fingerprint feature points and the directions of the fingerprint lines extending from the fingerprint feature points from ground truth fingerprint feature point information; and a first learning means for updating the parameters of a model that estimates the first estimation heatmap based on the first estimation heatmap and the first ground truth heatmap.

Images