Feature extraction device, object re-identification device, learning device, and feature extraction method

Abstract

A feature extraction device (10) is configured to include: an object image acquisition unit (12) that acquires information indicating an image in which a re-identification-target object is captured, the information being obtained from information indicating a reference image which is captured by an imaging device (2) and in which the object and a background of the object are captured; a background image acquisition unit (13) that acquires information indicating an image in which the background is captured, from the information indicating the image captured by the imaging device (2); and a feature amount generation unit (14) that integrates a feature amount of the object extracted on the basis of the information acquired by the object image acquisition unit (12) and a feature amount of the background extracted on the basis of the information acquired by the background image acquisition unit (13), and generates an integrated feature amount for re-identifying the object.

Description

Feature extraction device, object identification device, learning device, and feature extraction method

[0001] The present disclosure relates to a feature extraction device, an object identification device, a learning device, and a feature extraction method.

[0002] Conventionally, as a technology for determining what an object shown in an image is, object identification technology (Object re-identification) is known. Object identification technology is used, for example, in person identification that determines whether a person shown in each of a plurality of images is the same person by referring to the similarity between the images.

[0003] Person identification is performed, for example, by the following procedure. First, the object identification device extracts the feature amount of each of the plurality of images to be compared by image processing such as a convolutional neural network (CNN). Next, the object identification device calculates the distance or similarity between the extracted feature amounts. As the distance in this case, for example, the Euclidean distance is used, and as the similarity, for example, the cosine similarity is used. Then, the object identification device compares the calculated distance or similarity with a preset distance threshold or similarity threshold, and if the calculated distance or similarity is less than the threshold, it determines that the persons shown in the respective images are the same person. On the other hand, if the calculated distance or similarity is greater than or equal to the threshold, the object identification device determines that the persons shown in the respective images are not the same person (are different persons).

[0004] In relation to such person identification, for example, in Non-Patent Document 1, a person identification method based on instance segmentation and pose estimation has been proposed. In this method, in person identification, in order to suppress the influence of information on the background shown behind the person, an instance segmentation and skeleton extraction process is performed on the image in which the person is shown, and a method of extracting feature amounts using the results as the input of a convolutional neural network has been proposed. In this method, existing instance segmentation and pose estimation technologies are used.

[0005] Person Re-Identification Based on Instance Segmentation and Pose Estimation” Xihua University, IEEE PRAI 2023.

[0006] Object identification technology is crucial in "computer vision," where computers understand information in images like humans do. However, it faces challenges such as reduced identification accuracy due to the influence of background information, including the clutter of the background in images and differences in backgrounds between images taken with different cameras.

[0007] In this regard, the method proposed in Non-Patent Document 1 mentioned above can address the above-mentioned issues by suppressing the influence of background information reflected behind the person. However, this method has the problem that, prior to the extraction of features that contribute to the accuracy of object identification, certain preprocessing such as instance segmentation and skeleton extraction is required, and the accuracy of the features extracted through such preprocessing depends on the accuracy of the preprocessing.

[0008] This disclosure was made to solve the above-mentioned problems and aims to provide a feature extraction device that can extract feature quantities that reduce the dependence on the accuracy of preprocessing compared to conventional methods, while suppressing the influence of information about the background of an object on the accuracy of identifying that object.

[0009] The feature extraction device according to this disclosure includes: an object image acquisition unit that acquires information indicating an image showing an object from a reference image captured by an imaging device, which shows the object to be identified and the background of the object; a background image acquisition unit that acquires information indicating an image showing the background from information indicating an image captured by an imaging device; and a feature quantity generation unit that integrates the feature quantities of the object extracted based on the information acquired by the object image acquisition unit and the feature quantities of the background extracted based on the information acquired by the background image acquisition unit to generate integrated feature quantities for identifying the object.

[0010] According to this disclosure, the above configuration makes it possible to extract feature quantities that reduce the dependence on the accuracy of preprocessing compared to conventional methods, while suppressing the influence of background information on the object on the accuracy of object identification.

[0011] This figure shows an example configuration of an object identification system, including a feature extraction device and an object identification device according to Embodiment 1. Figures 2A and 2B are flowcharts showing an example of operation of the feature extraction device according to Embodiment 1. This is a flowchart showing an example of operation of the object identification device according to Embodiment 1. This figure is for explaining the effect of the feature extraction device according to Embodiment 1. This is a figure for explaining a comparative example and for explaining the flow of a conventional person identification process. This figure shows an example configuration of a feature extraction device that implements method 2 in another example of Embodiment 1. Figures 7A and 7B are diagrams showing an example of hardware configuration of the feature extraction device according to Embodiment 1. This figure shows an example configuration of a learning device according to Embodiment 1. This figure shows an example configuration of an object identification system, including a feature extraction device and an object identification device according to Embodiment 2. Figures 10A and 10B are flowcharts showing an example of operation of the feature extraction device according to Embodiment 2. This figure is for explaining an example of processing of the object feature extraction unit in Embodiment 2. This figure shows an example configuration of a learning device according to Embodiment 3. This figure is for explaining an example of operation of the learning device according to Embodiment 3.

[0012] The embodiments will now be described in detail with reference to the drawings. Embodiment 1. Figure 1 is a diagram showing an example of the configuration of an object identification system 1, which includes a feature extraction device 10 and an object identification device 50 according to Embodiment 1. As shown in Figure 1, the object identification system 1 is configured to include an imaging device 2, a feature extraction device 10, and an object identification device 50. In Figure 1, the imaging device 2 and the feature extraction device 10, and the feature extraction device 10 and the object identification device 50 are connected in a communicative manner by communication lines (not shown).

[0013] The feature extraction device 10 generates integrated features that the object identification device 50 uses to identify objects, based on information representing the image captured by the imaging device 2. The object identification device 50 uses the integrated features generated by the feature extraction device 10 to identify objects in the image captured by the imaging device 2. For the sake of clarity, in the following explanation, the object that the object identification device 50 identifies using the integrated features generated by the feature extraction device 10 will be assumed to be a person. Integrated features will be explained later.

[0014] <Imaging Device 2> The imaging device 2 is composed of, for example, a camera. The imaging device 2 captures images of the person to be identified and their background in a time series. In the following description, images of the person to be identified and their background are referred to as "reference images". The video information is composed of a set of reference images captured in a time series and a set of images that do not show the person to be identified but only show the background, i.e., images other than reference images. The imaging device 2 outputs this video information, including the set of reference images and the set of images other than reference images, to the feature extraction device 10.

[0015] <Feature Extraction Device 10> The feature extraction device 10 is configured to include, for example, a video information storage unit 11, an object image acquisition unit 12, a background image acquisition unit 13, and a feature quantity generation unit 14, as shown in Figure 1.

[0016] The video information storage unit 11 is configured, for example, with an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The video information storage unit 11 acquires video information output from the imaging device 2 and stores the acquired video information.

[0017] The object image acquisition unit 12 acquires information indicating an image showing a person, which is obtained from information indicating a reference image captured by the imaging device 2 that shows the person to be identified and the background of the person.

[0018] For example, the object image acquisition unit 12 acquires information indicating one reference image (hereinafter referred to as "reference image information") from the video information stored in the video information storage unit 11. When the object image acquisition unit 12 acquires one reference image information from the video information stored in the video information storage unit 11, it applies known person detection processing to the reference image indicated by the acquired reference image information and detects the image portion in which a person is visible from the reference image. As a result, the object image acquisition unit 12 acquires information indicating an image in which a person is visible (hereinafter referred to as "person image information"). At this time, the object image acquisition unit 12 also generates region information that can identify the region of the image portion in the reference image in which a person is visible.

[0019] The object image acquisition unit 12 outputs the acquired person image information to the feature generation unit 14. The object image acquisition unit 12 also outputs the generated region information to the background image acquisition unit 13. The person image information may include region information that can identify the area of ​​the image where the person is visible in the reference image. If the person image information includes region information, the object image acquisition unit 12 may output the acquired person image information to the background image acquisition unit 13 and the feature generation unit 14.

[0020] In this example, the object image acquisition unit 12 acquires reference image information from the video information stored in the video information storage unit 11, and then applies a known person detection process to the reference image indicated by the acquired reference image information to acquire person image information. However, the method by which the object image acquisition unit 12 acquires person image information is not limited to this. For example, the object image acquisition unit 12 may acquire person image information that has been manually acquired by the user of the feature extraction device 10 (hereinafter simply referred to as "user").

[0021] In this case, the user obtains one reference image from the video information stored in the video information storage unit 11, and then manually detects a person in the reference image indicated by the obtained reference image, thereby obtaining person image information. The user also generates region information that can identify the area of ​​the image in the reference image where the person is visible. The user then operates the input unit (not shown) of the feature extraction device 10 and inputs the obtained person image information and the generated region information into the object image acquisition unit 12.

[0022] Alternatively, the object image acquisition unit 12 may acquire person image information acquired by an external device (not shown) other than the feature extraction device 10 from that external device. In this case, the external device acquires one reference image from the video information stored in the video information storage unit 11, and acquires person image information by applying the known person detection process described above to the reference image indicated by the acquired reference image information. The external device also generates region information that can identify the area of ​​the image portion in the reference image where the person is visible. The external device can then input the acquired person image information and the generated region information to the object image acquisition unit 12, for example, via a network (not shown) connecting the external device and the feature extraction device 10.

[0023] The background image acquisition unit 13 acquires information indicating the background from the information indicating the image captured by the imaging device 2.

[0024] For example, the background image acquisition unit 13 refers to the video information storage unit 11 and acquires information (hereinafter referred to as "other image information") from the video information stored in the video information storage unit 11 that indicates an image captured by the imaging device 2, in a field of view corresponding to the field of view when the above-mentioned reference image was captured, but at a different time than when the above-mentioned reference image was captured.

[0025] As mentioned above, the reference image contains the person to be identified, but it is assumed that images taken at a different time than the reference image may not contain that person. Therefore, the background image acquisition unit 13 may acquire information as other image information that indicates images taken at a different time than the reference image and that do not contain the person to be identified. In this case, the background image acquisition unit 13 may, for example, apply a known person detection process to the information to be acquired to confirm in advance that the image indicated by the information does not contain the person to be identified before acquiring the information. Alternatively, the user may, for example, visually confirm in advance that the image indicated by the information does not contain the person to be identified, and the background image acquisition unit 13 may acquire information indicating the image that has been confirmed by the user not to contain the person.

[0026] Furthermore, the background image acquisition unit 13 acquires region information from the object image acquisition unit 12 that allows it to identify the region of the image where a person is visible in the reference image. After acquiring region information from the object image acquisition unit 12 and other image information from the video information storage unit 11, the background image acquisition unit 13 uses this acquired information to cut out the same area in the reference image where a person is visible from the image indicated by the other image information. As a result, the background image acquisition unit 13 obtains an image showing the background in the same area as where a person is visible in the reference image. Thus, the background image acquisition unit 13 acquires information indicating the image showing the background (hereinafter referred to as "background image information"). The background image acquisition unit 13 outputs the acquired background image information to the feature generation unit 14.

[0027] The background image acquisition unit 13 may also acquire background image information acquired by an external device (not shown) other than the feature extraction device 10. In this case, the external device acquires region information from the object image acquisition unit 12 and other image information from the video information storage unit 11. Once the external device has acquired region information from the object image acquisition unit 12 and other image information from the video information storage unit 11, it uses this acquired information to cut out the same area from the other image information that is the same as the area in the reference image where a person is visible. As a result, the external device obtains an image showing the background at the same area as the area in the reference image where a person is visible. Thus, the external device acquires information indicating the image showing the background (background image information). The external device can then input the acquired background image information to the background image acquisition unit 13, for example, via a network (not shown) connecting the external device and the feature extraction device 10.

[0028] The feature generation unit 14 integrates the features of a person extracted based on the person image information acquired by the object image acquisition unit 12 and the features of the background extracted based on the background image information acquired by the background image acquisition unit 13 to generate integrated features for identifying a person.

[0029] The processing performed by the feature generation unit 14 can be broadly divided into three processes: extraction of human features based on human image information acquired by the object image acquisition unit 12, extraction of background features based on background image information acquired by the background image acquisition unit 13, and integration of human features and background features. To accommodate these three processes, the feature generation unit 14 is configured to include an object feature extraction unit 141, a background feature extraction unit 142, and a feature integration unit 143.

[0030] The object feature extraction unit 141 extracts human features based on human image information acquired by the object image acquisition unit 12. The object feature extraction unit 141 is composed of a single convolutional neural network (CNN) that is pre-trained to take human image information as input and extract human features based on the input human image information.

[0031] The background feature extraction unit 142 extracts background features based on the background image information acquired by the background image acquisition unit 13. The background feature extraction unit 142 is composed of, for example, a single convolutional neural network (CNN) that has been pre-trained to take background image information as input and extract background features based on the input background image information.

[0032] The feature integration unit 143 integrates the human feature quantities extracted by the object feature quantity extraction unit 141 and the background feature quantities extracted by the background feature quantity extraction unit 142 to generate integrated feature quantities. The feature integration unit 143 outputs the generated integrated feature quantities to the object identification device 50.

[0033] Furthermore, when generating integrated features, the feature integration unit 143 may assign predetermined weights to at least one of the person features and the background features, and then integrate both sets of features. In this case, the features to which weights are assigned, and the specific values ​​of the weights, can be set by the user as appropriate. This allows the feature integration unit 143 to generate integrated features that better reflect either the person features or the background features.

[0034] Furthermore, when the feature integration unit 143 integrates the features of a person and the features of the background, it may integrate the feature vector of the person and the feature vector of the background, and then input the integrated vector into a layer of an arbitrary neural network (for example, a fully connected layer) to generate integrated features.

[0035] In the above description, an example was given in which the feature extraction device 10 is configured to include a video information storage unit 11. However, the video information storage unit 11 is not limited to this and may be provided in, for example, an external device to the feature extraction device 10.

[0036] <Object Identification Device 50> The object identification device 50 is configured to include, for example, a feature storage unit 51, a feature acquisition unit 52, and an object identification unit 53, as shown in Figure 1.

[0037] The feature storage unit 51 is configured, for example, with an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The feature storage unit 51 stores the integrated features output from the feature integration unit 143 of the feature extraction device 10.

[0038] The feature acquisition unit 52 acquires the integrated features stored in the feature storage unit 51.

[0039] The object identification unit 53 uses the integrated features obtained by the feature acquisition unit 52 to identify a person. For example, the object identification unit 53 calculates the distance or similarity between two integrated features obtained by the feature acquisition unit 52. In this case, for example, the Euclidean distance may be used as the distance, and for example, the cosine similarity may be used as the similarity. The object identification unit 53 then compares the calculated distance or similarity with a pre-set threshold for distance or similarity. If the calculated distance or similarity is less than the threshold, it determines that the person from whom the two integrated features were extracted, i.e., the person in the two reference images, is the same person. On the other hand, if the calculated distance or similarity is greater than or equal to the threshold, the object identification unit 53 determines that the person from whom the two integrated features were extracted is not the same person (they are different people). The object identification unit 53 may also output the person identification result to an output unit (not shown), such as a display.

[0040] In the above explanation, we described an example in which the object identification unit 53 identifies two people using two integrated features. However, the object identification unit 53 is not limited to this and may identify three or more people using three or more integrated features.

[0041] Furthermore, the above description described an example in which the object identification device 50 is configured to include a feature storage unit 51. However, the feature storage unit 51 is not limited to this and may be provided in an external device such as inside the feature extraction device 10.

[0042] In addition, when calculating the similarity between two integrated feature amounts acquired by the feature amount acquisition unit 52, the object identification unit 53 may calculate the similarity between the two integrated feature amounts using a learned model. Further, the learned model for calculating the similarity may be generated by a learning device. The learning device will be described later.

[0043] Next, an operation example of the feature extraction device 10 shown in FIG. 1 will be described. FIGS. 2A and 2B are flowcharts showing an operation example of the feature extraction device 10.

[0044] First, the object image acquisition unit 12 acquires information indicating an image in which a person is reflected, which is a reference image captured by the imaging device 2 and is obtained from a reference image in which the person to be identified and the background of the person are reflected (step ST1).

[0045] For example, the object image acquisition unit 12 acquires information (reference image information) indicating one reference image from the video information stored in the video information storage unit 11. When the object image acquisition unit 12 acquires one reference image information from the video information storage unit 11, the object image acquisition unit 12 performs known person detection processing on the reference image indicated by the acquired reference image information, and detects an image portion of a location where a person is reflected from the reference image. Thereby, the object image acquisition unit 12 acquires information (person image information) indicating an image in which a person is reflected. At this time, the object image acquisition unit 12 also generates region information that can specify the region of the image portion of the location where a person is reflected in the reference image.

[0046] The object image acquisition unit 12 outputs the acquired person image information to the feature amount generation unit 14. Further, the object image acquisition unit 12 outputs the generated above-described region information to the background image acquisition unit 13.

[0047] Next, the background image acquisition unit 13 acquires information indicating an image in which the background is reflected from the information indicating the image captured by the imaging device 2 (step ST2).

[0048] For example, the background image acquisition unit 13 refers to the video information storage unit 11, and from the video information storage unit 11, an image captured by the imaging device 2, and at an angle of view corresponding to the angle of view when the above-mentioned reference image was captured, information (other image information) indicating an image captured at a time different from the time when the above-mentioned reference image was captured is acquired.

[0049] Here, as described above, the reference image shows the person to be identified, but it is assumed that the images captured at times different from the time when the reference image was captured include images in which the person is not shown. Therefore, the background image acquisition unit 13 may acquire, as other image information, information indicating an image captured at a time different from the time when the reference image was captured and in which the person to be identified is not shown.

[0050] Further, the background image acquisition unit 13 acquires, from the object image acquisition unit 12, region information that can specify the region of the image portion where the person appears in the reference image. When the background image acquisition unit 13 acquires the region information from the object image acquisition unit 12 and acquires the other image information from the video information storage unit 11, based on these acquired information, the same location as the location where the person appears in the reference image is cut out from the image indicated by the other image information. Thereby, the background image acquisition unit 13 obtains an image in which the background at the same location as the location where the person appears in the reference image is shown. And thereby, the background image acquisition unit 13 acquires information (background image information) indicating the image in which the background is shown. The background image acquisition unit 13 outputs the acquired background image information to the feature amount generation unit 14.

[0051] Next, the feature amount generation unit 14 acquires the person image information from the object image acquisition unit 12 and the background image information from the background image acquisition unit 13. Then, the feature amount generation unit 14 integrates the feature amounts of the person extracted based on the person image information acquired from the object image acquisition unit 12 and the feature amounts of the background extracted based on the background image information acquired from the background image acquisition unit 13, and generates an integrated feature amount for identifying the person (step ST3).

[0052] Step ST3, as shown in Figure 2B, consists of the following substeps ST3-1 to ST3-3. Specifically, the object feature extraction unit 141 extracts human feature quantities based on human image information acquired from the object image acquisition unit 12 (substep ST3-1).

[0053] The background feature extraction unit 142 extracts background features based on the background image information acquired from the background image acquisition unit 13 (substep ST3-2).

[0054] The feature integration unit 143 integrates the features of the person extracted by the object feature extraction unit 141 and the features of the background extracted by the background feature extraction unit 142 to generate integrated features (substep ST3-3). For example, the feature integration unit 143 generates integrated features by integrating the features of the person and the features of the background using methods such as subtraction, addition, or concatenation.

[0055] The feature integration unit 143 can, for example, generate integrated features that more strongly emphasize the differences between the person and the background, or the change from the person to the background, when integrating by subtraction. Furthermore, the feature integration unit 143 can generate integrated features that are the sum of the features of the person and the features of the background when integrating by addition. Also, the feature integration unit 143 can generate integrated features that retain the features of the person and the features of the background as they are, and that are of a higher dimension than the individual features when integrating by concatenation. In substep ST3-3, the feature integration unit 143 may assign a predetermined weight to at least one of the features of the person and the features of the background before integrating both sets of features.

[0056] In the flowchart shown in Figure 2, substep ST3-2 is executed after substep ST3-1, but the processing order of each substep is not limited to the example above. For example, substep ST3-2 may be performed before substep ST3-1, or substeps ST3-1 and ST3-2 may be performed in parallel.

[0057] Next, an example of the operation of the object identification device 50 shown in Figure 1 will be described. Figure 3 is a flowchart illustrating an example of the operation of the object identification device 50.

[0058] First, the feature acquisition unit 52 acquires the integrated features stored in the feature storage unit 51 (step ST11).

[0059] Next, the object identification unit 53 uses the integrated features acquired by the feature acquisition unit 52 to identify the person (step ST12).

[0060] Next, the effects of the feature extraction device 10 according to Embodiment 1 will be explained in comparison with the comparative example. Figure 4 is a diagram illustrating the effects of the feature extraction device 10 according to Embodiment 1. Figure 5 is a diagram illustrating the comparative example and illustrates the flow of a conventional person identification process.

[0061] As shown in Figure 5, in the comparative example, two images A and B, both depicting the same person, are input to the conventional feature extraction device 100. In Figure 5, these two images are labeled "Person Image A" and "Person Image B".

[0062] The feature extraction device 100 includes an object feature extraction unit 200. The object feature extraction unit 200 is configured by a pre-trained convolutional neural network that takes, for example, person image information as input and extracts the features of the person depicted in the image based on the input person image information.

[0063] When a person image A is input to the feature extraction device 100, the object feature extraction unit 200 extracts the feature quantities of the person depicted in person image A based on the person image information representing person image A. In Figure 5, these feature quantities are labeled "person feature quantity A". Similarly, when a person image B is input to the feature extraction device 100, the object feature extraction unit 200 extracts the feature quantities of the person depicted in person image B based on the person image information representing person image B. In Figure 5, these feature quantities are labeled "person feature quantity B". The object feature extraction unit 200 outputs person feature quantity A and person feature quantity B to the object identification device 500 according to a conventional configuration.

[0064] Each object identification device 500 is comprised of a feature quantity storage unit 501 (not shown), a feature quantity acquisition unit 502, and an object identification unit 503.

[0065] The feature storage unit 501 is configured, for example, with an HDD or SSD, and stores the person feature A and person feature B output from the object feature extraction unit 200 of the feature extraction device 100.

[0066] The feature acquisition unit 502 acquires person feature A and person feature B stored in the feature storage unit 501.

[0067] The object identification unit 503 uses person feature A and person feature B acquired by the feature acquisition unit 502 to identify a person. For example, the object identification unit 503 calculates the distance or similarity between the two feature quantities, compares the calculated distance or similarity with a pre-set distance threshold or similarity threshold, and determines that the person in each person image is the same person if the calculated distance or similarity is less than the threshold. On the other hand, if the calculated distance or similarity is greater than or equal to the threshold, the object identification unit 503 determines that the person in each person image is not the same person (they are different people).

[0068] In this case, the object identification unit 503 should ideally determine that the person in person image A and person image B is the same person. However, if, for example, a background is visible behind the person in person image A and person image B, and the backgrounds behind the person in both images are significantly different, the object identification unit 503 may mistakenly determine that the person in both images is not the same person. This is because, when the feature extraction device 100 extracts person feature A and person feature B, it extracts person feature A from person image A and person feature B from person image B without particularly considering the influence of the background. As a result, information from the background becomes noise and gets mixed into person feature A and person feature B, causing a large discrepancy in the distance or similarity between these two feature quantities.

[0069] In contrast, in the feature extraction device 10 according to Embodiment 1, as shown in Figure 4, not only information indicating two person images A and B in which the same person is depicted is input to the feature generation unit 14, but also information indicating images showing the background at the same location as the person in these images. In Figure 4, these background images are labeled "background image A" and "background image B".

[0070] When information indicating a person image A and a background image A is input to the feature generation unit 14, the object feature extraction unit 141 extracts the features of the person depicted in person image A based on the person image information indicating person image A. In Figure 4, these features are denoted as "person feature A".

[0071] Furthermore, the background feature extraction unit 142 extracts background features from background image A based on background image information representing background image A. In Figure 4, these features are referred to as "background feature A".

[0072] The feature integration unit 143 then integrates the person feature A extracted by the object feature extraction unit 141 and the background feature A extracted by the background feature extraction unit 142 to generate an integrated feature. In Figure 4, this feature is referred to as "integrated feature A".

[0073] Furthermore, when information indicating the person image B and the background image B is input to the feature generation unit 14, the object feature extraction unit 141, the background feature extraction unit 142, and the feature integration unit 143 generate integrated feature B using the same procedure as the integrated feature A generation procedure described above. The feature integration unit 143 outputs the generated integrated feature A and integrated feature B to the object identification device 50 according to Embodiment 1.

[0074] The object identification device 50 performs person identification using integrated feature A and integrated feature B output from the feature generation unit 14.

[0075] Specifically, the feature storage unit 51 stores the integrated feature A and integrated feature B output from the feature integration unit 143 of the feature extraction device 10. The feature acquisition unit 52 acquires the integrated feature A and integrated feature B stored in the feature storage unit 51. The object identification unit 53 uses the integrated feature A and integrated feature B acquired by the feature acquisition unit 52 to identify a person.

[0076] In this case, the object identification unit 53 is less likely than in the comparative example to misidentify that the people in the two images are not the same person, even if, for example, the background is reflected behind the person in person image A and person image B, and the backgrounds reflected behind the person in both images are significantly different. This is because the integrated feature quantities A and B generated by the feature extraction device 10 are an integration of person features and background features; in other words, they are feature quantities that take into account the influence of the background in both person image A and person image B, so the information contained in the background is incorporated into the integrated feature quantities as features rather than as noise.

[0077] As a result, the feature extraction device 10 according to Embodiment 1 can generate feature quantities for person identification that are robust to the influence of variations in background information, such as differences in the time of day when a reference image showing a person and background is captured, or differences in lighting conditions. In other words, the feature extraction device 10 according to Embodiment 1 can extract feature quantities in which the influence of information about the person's background on the accuracy of person identification is suppressed. Furthermore, the object identification device 50 according to Embodiment 1 can perform person identification with greater accuracy than conventional methods by using the integrated feature quantities generated by the feature extraction device 10 to perform person identification.

[0078] Furthermore, in the feature extraction device 10 according to Embodiment 1, when extracting background features, the background image acquisition unit 13 acquires background image information from information indicating the image captured by the imaging device 2, and the background feature extraction unit 142 extracts background features based on the background image information acquired from the background image acquisition unit 13. As a result, the feature extraction device 10 according to Embodiment 1 eliminates the need for predetermined preprocessing such as instance segmentation and skeleton extraction processing that were previously required, and when extracting features for identifying people, it is possible to reduce the workload of the preprocessing itself and the degree of dependence on the accuracy of the preprocessing compared to conventional methods.

[0079] <Other Examples of Embodiment 1> Next, other examples of the feature extraction device 10 according to Embodiment 1 will be described.

[0080] Up to this point, the explanation described an example in which the background image acquisition unit 13 acquires information indicating an image showing the background at the same location as the location where the person is visible (background image information) based on region information acquired from the object image acquisition unit 12 and other image information acquired from the video information storage unit 11, that is, information indicating an image that was taken at a different time than when the reference image was taken and does not show a person.

[0081] Here, the background image acquisition unit 13 may acquire background image information using, for example, an image taken at a different time than when the reference image was taken, and which does not contain any people, such as an image taken immediately before the reference image was taken (the frame immediately preceding the reference image). However, in this case, the background image indicated by the acquired background image information may not perfectly match the person image indicated by the person image information acquired by the object image acquisition unit 12, and may be a background image containing noise.

[0082] Furthermore, the background image acquisition unit 13 may acquire background image information using an image taken at a different time than the time the reference image was taken, and which does not contain any people. However, in this case, the background image indicated by the acquired background image information may not perfectly match the person image indicated by the person image information acquired by the object image acquisition unit 12, and may be a background image containing noise.

[0083] Therefore, in order to mitigate the generation of noise as described above, the background image acquisition unit 13 may perform processing by, for example, the following method.

[0084] <Method 1> For example, the background image acquisition unit 13 acquires multiple images taken at a different time than the time the reference image was taken, from the video information stored in the video information storage unit 11.

[0085] The background image acquisition unit 13 then calculates, for example, the average or median of the pixel values ​​of the pixels contained in each of these multiple images, and obtains one image represented by the calculated average or median. Alternatively, the background image acquisition unit 13 obtains a background image using a method that performs person detection on multiple images and extracts images in which no person was detected, or a method that creates a background model and generates a background (background subtraction method, for example, a Gaussian mixture model). In this way, the background image acquisition unit 13 obtains one piece of background image information. The background image acquisition unit 13 outputs this background image information to the feature generation unit 14.

[0086] The object feature extraction unit 141 of the feature generation unit 14 extracts human features based on human image information output from the object image acquisition unit 12. The background feature extraction unit 142 extracts background features based on background image information output from the background image acquisition unit 13. The feature integration unit 143 then generates integrated features by integrating the human features extracted by the object feature extraction unit 141 and the background features extracted by the background feature extraction unit 142.

[0087] <Method 2> Figure 6 shows an example configuration of a feature extraction device 10 for implementing Method 2. As shown in Figure 6, the feature extraction device 10 may have multiple background image acquisition units 13, or it may have the same number of background feature extraction units 142 as the background image acquisition units 13. For example, each background image acquisition unit 13 acquires images taken at a different time than the time the reference image was taken from the video information stored in the video information storage unit 11, similar to Method 1. In this case, the images acquired by each background image acquisition unit 13 do not necessarily have to be temporally consecutive. As a result, each background image acquisition unit 13 obtains multiple pieces of background image information as a whole. Each background image acquisition unit 13 outputs the background image information to the feature generation unit 14.

[0088] The object feature extraction unit 141 of the feature generation unit 14 extracts human feature quantities based on the human image information output from the object image acquisition unit 12. Each background feature extraction unit 142 extracts background feature quantities based on the background image information output from each background image acquisition unit 13. The feature integration unit 143 then generates multiple integrated feature quantities by individually integrating the human feature quantities extracted by the object feature extraction unit 141 and the background feature quantities extracted by each background feature extraction unit 142.

[0089] The background image acquisition unit 13 can obtain background image information with less noise than Embodiment 1 by using multiple images taken at different times than the time the reference image was taken as input, as described in Methods 1 and 2 above. In other words, the background image acquisition unit 13 can acquire background image information that can improve the accuracy of person identification by using Methods 1 and 2 above. As a result, the feature extraction device 10 according to another example of Embodiment 1 can generate integrated features that enable more accurate person identification than the feature extraction device 10 according to Embodiment 1.

[0090] Next, with reference to Figure 7, an example of the hardware configuration of the feature extraction device 10 according to Embodiment 1 will be described. The functions of the object image acquisition unit 12, the background image acquisition unit 13, and the feature quantity generation unit 14 in the feature extraction device 10 are realized by a processing circuit. The processing circuit may be dedicated hardware as shown in Figure 7A, or it may be a CPU (Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 62 that executes a program stored in memory 63, as shown in Figure 7B.

[0091] If the processing circuit is dedicated hardware, the processing circuit 61 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processing circuit 61 may implement the functions of each part of the object image acquisition unit 12, background image acquisition unit 13, and feature quantity generation unit 14 individually, or it may implement the functions of each part together in the processing circuit 61.

[0092] When the processing circuit is a CPU 62, the functions of the object image acquisition unit 12, the background image acquisition unit 13, and the feature generation unit 14 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 63. The processing circuit realizes the functions of each unit by reading and executing the programs stored in memory 63. In other words, the feature extraction device 10 is equipped with memory for storing programs that, when executed by the processing circuit, result in the execution of each step shown in Figure 2, for example. These programs can also be said to cause the computer to execute the procedures and methods of the object image acquisition unit 12, the background image acquisition unit 13, and the feature generation unit 14. Here, the memory 63 can include, for example, non-volatile or volatile semiconductor memory such as RAM (RAM Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, or DVDs (Digital Versatile Discs).

[0093] Furthermore, the functions of the object image acquisition unit 12, background image acquisition unit 13, and feature generation unit 14 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the function of the object image acquisition unit 12 can be implemented by a processing circuit as dedicated hardware, while the functions of the background image acquisition unit 13 and the feature generation unit 14 can be implemented by a processing circuit reading and executing a program stored in memory 63.

[0094] Thus, the processing circuit can realize each of the above-mentioned functions through hardware, software, firmware, or a combination thereof.

[0095] Next, the learning device described above will be explained. The learning device is a learning model used by the object identification unit 53, and generates a trained model for calculating the similarity between multiple (for example, two) integrated features acquired by the feature acquisition unit 52.

[0096] Figure 8 shows an example of the configuration of the learning device 70 according to Embodiment 1. For example, the learning device 70 is composed of a learning data storage unit 11', a feature extraction device 10 according to Embodiment 1 which includes functional units other than the video information storage unit 11, a task loss calculation unit 71, a parameter update unit 72, and a trained model storage unit 73.

[0097] The learning data storage unit 11' is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The learning data storage unit 11' is a functional unit that includes the video information storage unit 11 provided in the feature extraction device 10 according to Embodiment 1. In other words, the learning data storage unit 11' stores video information similar to that stored by the video information storage unit 11 in Embodiment 1. In addition to the video information, the learning data storage unit 11' also stores annotation data (for example, rectangular data of the person region and person ID, etc.) that the task loss calculation unit 71 refers to when calculating the task loss.

[0098] The feature extraction device 10's feature quantity integration unit 143 outputs the generated integrated features to the task loss calculation unit 71.

[0099] The task loss calculation unit 71 obtains integrated features from the feature integration unit 143. Based on the integrated features obtained from the feature integration unit 143 and the annotation data stored in the training data storage unit 11', the task loss calculation unit 71 calculates a task loss to calculate, for example, the similarity between two integrated features. In calculating this task loss, for example, a known loss function is used to measure the difference between the ground truth similarity (label) between integrated feature A and integrated feature B and the similarity calculated by the learning model.

[0100] In this example, the task loss calculation unit 71 calculates a task loss for calculating the similarity between two integrated features based on integrated features obtained from the feature integration unit 143. However, the task loss calculation unit 71 is not limited to this example and may use a known loss function to calculate a task loss for calculating the similarity between, for example, three or more integrated features. The task loss calculation unit 71 outputs the calculated task loss to the parameter update unit 72.

[0101] The parameter update unit 72 obtains the task loss from the task loss calculation unit 71. Based on the task loss obtained from the task loss calculation unit 71, the parameter update unit 72 updates the parameters of the learning model. For example, the parameter update unit 72 updates the parameters of the learning model in a direction that minimizes the task loss obtained from the task loss calculation unit 71.

[0102] The calculation of task loss by the task loss calculation unit 71 and the updating of the parameters of the learned model by the parameter update unit 72 may be performed multiple times. As a result, the parameter update unit 72 obtains a trained model that the object identification unit 53 uses to calculate the similarity between multiple integrated features. The parameter update unit 72 saves the obtained trained model in the trained model storage unit 73.

[0103] The trained model storage unit 73 is configured, for example, as an HDD or SSD. The trained model storage unit 73 stores the trained model obtained by the parameter update unit 72.

[0104] Thus, the learning device 70 is configured to include a learning data storage unit 11', a feature extraction device 10 including an object image acquisition unit 12, a background image acquisition unit 13, and a feature quantity generation unit 14, a task loss calculation unit 71, a parameter update unit 72, and a trained model storage unit 73, thereby enabling the generation of a trained model used by the object identification unit 53 for calculating the similarity between multiple integrated features. The trained model storage unit 73 may be provided outside the learning device 70.

[0105] As described above, according to this embodiment 1, the feature extraction device 10 includes: an object image acquisition unit 12 that acquires information indicating an image showing an object from a reference image captured by the imaging device 2, which shows the object to be identified and the background of the object; a background image acquisition unit 13 that acquires information indicating an image showing the background from information indicating an image captured by the imaging device 2; and a feature quantity generation unit 14 that integrates the feature quantities of the object extracted based on the information acquired by the object image acquisition unit 12 and the feature quantities of the background extracted based on the information acquired by the background image acquisition unit 13 to generate integrated feature quantities for identifying the object. As a result, the feature extraction device 10 according to embodiment 1 can extract feature quantities that reduce the degree of dependence on the accuracy of preprocessing compared to conventional methods, while suppressing the influence of information on the background of the object on the accuracy of identifying the object.

[0106] Furthermore, the feature generation unit 14 includes an object feature extraction unit 141 that extracts object features based on information acquired by the object image acquisition unit 12, a background feature extraction unit 142 that extracts background features based on information acquired by the background image acquisition unit 13, and a feature integration unit 143 that integrates the object features extracted by the object feature extraction unit 141 and the background features extracted by the background feature extraction unit 142 to generate integrated features. As a result, the feature extraction device 10 according to Embodiment 1 can generate integrated features that combine the features of objects and the features of the background.

[0107] Furthermore, the feature integration unit 143 assigns a predetermined weight to at least one of the object features and the background features, and then integrates both sets of features. As a result, the feature extraction device 10 according to Embodiment 1 can generate integrated features that better reflect either the person features or the background features.

[0108] Furthermore, the background image acquisition unit 13 acquires information indicating images showing the background based on information indicating multiple images taken by the imaging device 2, where the field of view corresponds to the field of view when the reference image was taken, but taken at a different time than when the reference image was taken. As a result, the feature extraction device 10 according to Embodiment 1 can suppress noise contained in images showing the background.

[0109] Furthermore, the background image acquisition unit 13 obtains multiple pieces of information indicating images showing the background of an object, based on information indicating multiple images taken at a different time than the time the reference image was taken. The feature quantity generation unit 14 then individually integrates the multiple background feature quantities extracted based on the information indicating multiple background images acquired by the background image acquisition unit 13 with the feature quantities of the object extracted based on the information indicating the object, thereby generating multiple integrated feature quantities. As a result, the feature extraction device 10 according to Embodiment 1 can suppress noise contained in images showing the background.

[0110] Furthermore, according to this embodiment 1, the object identification device 50 is configured to include a feature acquisition unit 52 that acquires integrated feature quantities generated by the feature extraction device 10, and an object identification unit 53 that performs object identification using the integrated feature quantities acquired by the feature acquisition unit 52. As a result, the object identification device 50 according to embodiment 1 can perform object identification with higher accuracy than object identification using feature quantities generated by conventional methods.

[0111] Furthermore, according to this embodiment 1, the learning device 70 is configured to include a feature extraction device 10 which includes an object image acquisition unit 12, a background image acquisition unit 13, and a feature generation unit 14; a task loss calculation unit 71 which calculates a task loss for calculating the similarity between a plurality of integrated features generated by the feature generation unit 14; and a parameter update unit 72 which obtains a trained model by updating the parameters of the learning model based on the task loss calculated by the task loss calculation unit 71. As a result, the learning device 70 can generate a trained model which is used by the object identification unit 53 to calculate the similarity between a plurality of integrated features.

[0112] Embodiment 2. Embodiment 1 described a feature extraction device 10 that can extract feature quantities that reduce the degree of dependence on the accuracy of preprocessing compared to conventional methods, while suppressing the influence of information about the background of the object on the identification accuracy of the object. Embodiment 2 describes a feature extraction device 10b that can extract integrated feature quantities that are expected to improve the object identification accuracy compared to the feature extraction device 10 of Embodiment 1.

[0113] Figure 9 shows an example configuration of an object identification system 1b, including a feature extraction device 10b and an object identification device 50 according to Embodiment 2. As shown in Figure 9, the example configuration of the feature extraction device 10b according to Embodiment 2 is the same as that of the feature extraction device 10 according to Embodiment 1 shown in Figure 1, but with the object image acquisition unit 12 changed to an object image acquisition unit 12b, the background image acquisition unit 13 changed to a background image acquisition unit 13b, the feature quantity generation unit 14 changed to a feature quantity generation unit 14b, and a weight map generation unit 15 added. The other components of the object identification system 1b according to Embodiment 2 are the same as those of the object identification system 1 according to Embodiment 1 shown in Figure 1, so the same reference numerals are used and their descriptions are omitted.

[0114] The object image acquisition unit 12b acquires information indicating an image containing a person (person image information) obtained from a reference image captured by the imaging device 2, which shows the person to be identified and the background of that person, similar to the object image acquisition unit 12 in Embodiment 1.

[0115] The object image acquisition unit 12b outputs the acquired person image information to the feature generation unit 14b and the weight map generation unit 15. The object image acquisition unit 12b also outputs region information that allows identification of the area of ​​the image containing the person in the reference image to the background image acquisition unit 13b.

[0116] The background image acquisition unit 13b, similar to the background image acquisition unit 13 in Embodiment 1, acquires information indicating the background image (background image information) from information indicating the image captured by the imaging device 2. The background image acquisition unit 13b outputs the acquired background image information to the feature generation unit 14b and the weight map generation unit 15.

[0117] The weight map generation unit 15 acquires person image information from the object image acquisition unit 12b and background image information from the background image acquisition unit 13b. The weight map generation unit 15 uses either or both of the person image information acquired from the object image acquisition unit 12b and the background image information acquired from the background image acquisition unit 13b to generate a map (hereinafter referred to as "weight map") in which the magnitude of values ​​differs depending on the region. For example, the weight map generation unit 15 generates information showing the difference between the person image information and the background image information and treats this as a weight map. The weight map generation unit 15 may also obtain a sampling map from the person image information or the background image information and treat this as a weight map. The weight map generation unit 15 outputs the generated weight map to the feature generation unit 14b.

[0118] The feature generation unit 14b acquires person image information from the object image acquisition unit 12b and background image information from the background image acquisition unit 13b. The feature generation unit 14b also acquires a weight map from the weight map generation unit 15. The feature generation unit 14b integrates the person features extracted based on the person image information acquired from the object image acquisition unit 12b, which are combined with the weight map acquired from the weight map generation unit 15, and the background features extracted based on the background image information acquired from the background image acquisition unit 13b, to generate an integrated feature for identifying a person. Alternatively, the feature generation unit 14b integrates the background features extracted based on the background image information acquired from the background image acquisition unit 13b, which are combined with the weight map acquired from the weight map generation unit 15, and the person features extracted based on the person image information acquired from the object image acquisition unit 12b, to generate an integrated feature for identifying a person. Alternatively, the feature generation unit 14b integrates the features of a person extracted based on the person image information acquired from the object image acquisition unit 12b, which are the features of a person that have been modified by the weight map acquired from the weight map generation unit 15, and the features of the background extracted based on the background image information acquired from the background image acquisition unit 13b, which are the features of the background that have been modified by the weight map acquired from the weight map generation unit 15, to generate integrated features for identifying a person.

[0119] Specifically, the object feature extraction unit 141b extracts human feature quantities that take into account the weight map obtained from the weight map generation unit 15, based on the human image information obtained from the object image acquisition unit 12b and the weight map obtained from the weight map generation unit 15.

[0120] The background feature extraction unit 142b extracts background features based on background image information acquired from the background image acquisition unit 13b, similar to the background feature extraction unit 142 in Embodiment 1.

[0121] The feature integration unit 143b integrates the human feature quantities extracted by the object feature quantity extraction unit 141b and the background feature quantities extracted by the background feature quantity extraction unit 142b to generate integrated feature quantities.

[0122] Alternatively, the object feature extraction unit 141b extracts human features based on human image information acquired from the object image acquisition unit 12b, similar to the object feature extraction unit 141 in Embodiment 1.

[0123] The background feature extraction unit 142b extracts background features that take into account the weight map obtained from the weight map generation unit 15, based on the background image information obtained from the background image acquisition unit 13b and the weight map obtained from the weight map generation unit 15.

[0124] The feature integration unit 143b integrates the human feature quantities extracted by the object feature quantity extraction unit 141b and the background feature quantities extracted by the background feature quantity extraction unit 142b to generate integrated feature quantities.

[0125] Alternatively, the object feature extraction unit 141b extracts human feature quantities that take into account the weight map obtained from the weight map generation unit 15, based on the human image information obtained from the object image acquisition unit 12b and the weight map obtained from the weight map generation unit 15.

[0126] The background feature extraction unit 142b extracts background features that take into account the weight map obtained from the weight map generation unit 15, based on the background image information obtained from the background image acquisition unit 13b and the weight map obtained from the weight map generation unit 15.

[0127] The feature integration unit 143b integrates the human feature quantities extracted by the object feature quantity extraction unit 141b and the background feature quantities extracted by the background feature quantity extraction unit 142b to generate integrated feature quantities.

[0128] Next, we will describe an example of the operation of the feature extraction device 10b shown in Figure 9. Figure 10 is a flowchart showing an example of the operation of the feature extraction device 10b. Here, as an example, we will describe an example of operation in which the feature quantity generation unit 14b integrates the feature quantities of a person extracted based on the person image information acquired from the object image acquisition unit 12b, which are the feature quantities of a person that take into account the weight map acquired from the weight map generation unit 15, and the feature quantities of the background extracted based on the background image information acquired from the background image acquisition unit 13b, in order to generate integrated feature quantities for identifying a person.

[0129] First, the object image acquisition unit 12b acquires information indicating the image containing the person (person image information) obtained from a reference image captured by the imaging device 2, which shows the person to be identified and the background of the person (step ST21). The object image acquisition unit 12b outputs the acquired person image information to the feature generation unit 14b and the weight map generation unit 15. The object image acquisition unit 12b also outputs region information that allows identification of the area of ​​the image portion in the reference image where the person is visible to the background image acquisition unit 13b.

[0130] Next, the background image acquisition unit 13b acquires information indicating the background image (background image information) from the information indicating the image captured by the imaging device 2, similar to the background image acquisition unit 13 in Embodiment 1 (step ST22). The background image acquisition unit 13b outputs the acquired background image information to the feature generation unit 14b and the weight map generation unit 15.

[0131] Next, the weight map generation unit 15 acquires person image information from the object image acquisition unit 12b and background image information from the background image acquisition unit 13b. The weight map generation unit 15 uses either or both of the person image information acquired from the object image acquisition unit 12b and the background image information acquired from the background image acquisition unit 13b to generate a map (weight map) with different magnitudes of values ​​depending on the region (step ST23). The weight map generation unit 15 outputs the generated weight map to the feature generation unit 14b.

[0132] Next, the feature generation unit 14b acquires person image information from the object image acquisition unit 12b and background image information from the background image acquisition unit 13b. The feature generation unit 14b also acquires a weight map from the weight map generation unit 15.

[0133] Then, the feature generation unit 14b integrates the person features extracted based on the person image information acquired from the object image acquisition unit 12b, which are the person features with the weight map acquired from the weight map generation unit 15 added, and the background features extracted based on the background image information acquired from the background image acquisition unit 13b, to generate integrated features for identifying the person (step ST24).

[0134] Step ST24, as shown in Figure 10B, is composed of the following substeps ST24-1 to ST24-3. Specifically, the object feature extraction unit 141b extracts the features of a person, which are the features of a person that take into account the weight map obtained from the weight map generation unit 15, based on the person image information obtained from the object image acquisition unit 12b and the weight map obtained from the weight map generation unit 15 (substep ST24-1).

[0135] Here, the processing of substep ST24-1 will be explained with reference to Figure 11. The object feature extraction unit 141b is composed of a single convolutional neural network that has been pre-trained to take, for example, person image information as input and extract person features based on the input person image information. Here, this convolutional neural network is also referred to as the "first convolutional neural network".

[0136] The object feature extraction unit 141b extracts the human feature PF by taking into account the weight map obtained from the weight map generation unit 15 and adding it to the feature map output from any layer of the first convolutional neural network.

[0137] The feature vectors (PF) of a person extracted in this way incorporate a weight map that masks the background in images containing a person, for example. Therefore, it can be expected that the feature vectors of a person extracted in this way have the influence of the background appropriately removed and capture only the features of the person.

[0138] Furthermore, when the object feature extraction unit 141b adds a weight map obtained from the weight map generation unit 15 to a feature map output from any layer of the first convolutional neural network, it may adjust the size of the weight map to match the size of the feature map before adding it to the feature map. This allows the object feature extraction unit 141b to appropriately incorporate the weight map into the feature map, enabling it to accurately extract the feature quantity PF of a person with the influence of the background removed.

[0139] Furthermore, when the object feature extraction unit 141b adds a weight map obtained from the weight map generation unit 15 to a feature map output from any layer of the first convolutional neural network, it may process the weight map as follows, rather than adding it directly. For example, if the weight map is information that shows the difference between person image information and background image information, the object feature extraction unit 141b inputs the weight map to a feature extraction unit different from itself and extracts background difference features, which are features of the weight map. This feature extraction unit is composed of, for example, a second convolutional neural network.

[0140] Furthermore, the object feature extraction unit 141b may extract the human feature quantity PF after incorporating the extracted background difference features into the feature map described above. If the feature extraction unit described above is composed of a second convolutional neural network, the second convolutional neural network is a network that has been pre-trained to extract background difference features, which are features of the weight map, in response to the input of the weight map. As a result, the object feature extraction unit 141b can appropriately incorporate the background difference features extracted from the weight map into the feature map, and can extract the human feature quantity PF, from which the influence of the background has been removed, with even greater accuracy.

[0141] Next, the background feature extraction unit 142b extracts background features based on the background image information acquired from the background image acquisition unit 13b, similar to the background feature extraction unit 142 in Embodiment 1 (substep ST24-2).

[0142] Then, the feature integration unit 143b integrates the human feature quantities extracted by the object feature quantity extraction unit 141b and the background feature quantities extracted by the background feature quantity extraction unit 142b to generate integrated feature quantities (substep ST24-3).

[0143] In this description, an example has been given in which the feature extraction device 10b according to Embodiment 2 executes substeps ST24-2 and ST24-3 after substep ST24-1. However, in Embodiment 2, it is assumed that the person feature quantity PF extracted in substep ST24-1 is already a feature quantity from which the influence of the background has been appropriately removed. Therefore, in Embodiment 2, if it is considered that the person feature quantity PF obtained in substep ST24-1 can be used to accurately identify a person, the processes in substeps ST24-2 and ST24-3 may be omitted. In that case, the background feature quantity extraction unit 142b and the feature quantity integration unit 143b may be omitted in the feature extraction device 10b according to Embodiment 2. However, if substeps ST24-2 and ST24-3 are executed in addition to substep ST24-1 in Embodiment 2, the resulting integrated feature quantity is expected to be a feature quantity that is robust to changes in the background and capable of performing even more accurate person identification.

[0144] In the second embodiment, if the background feature extraction unit 142b and the feature integration unit 143b are omitted, the feature extraction device 10b is configured as follows, for example. (Example Configuration) A feature extraction device 10b comprising: an object image acquisition unit 12b that acquires information indicating an image showing the object, obtained from a reference image captured by an imaging device 2 which shows the object to be identified and the background of the object; a background image acquisition unit 13b that acquires information indicating an image showing the background from information indicating an image captured by the imaging device 2; a weight map generation unit 15 that generates a weight map, which is a map with different magnitudes of values ​​depending on the region, using either or both of the information indicating an image showing the object acquired by the object image acquisition unit 12b and the information indicating an image showing the background acquired by the background image acquisition unit 13b; and an object feature extraction unit 141b that extracts feature quantities of the object, which are the feature quantities of the object that take into account the weight map generated by the weight map generation unit 15, based on the information acquired by the object image acquisition unit 12b and the information generated by the weight map generation unit 15.

[0145] Furthermore, in Embodiment 2, a learning device similar to the learning device 70 described in Embodiment 1 may be configured. In that case, the learning device according to Embodiment 2 is configured to include, for example, a learning data storage unit 11' that includes a video information storage unit 11, a feature extraction device 10b that includes an object image acquisition unit 12b, a background image acquisition unit 13b, and a feature quantity generation unit 14b, a task loss calculation unit 71, a parameter update unit 72, and a trained model storage unit 73.

[0146] As described above, according to this second embodiment, the feature extraction device 10b includes a weight map generation unit 15 that generates a weight map with different magnitudes of values ​​depending on the region, using either or both of the information indicating an image in which an object is captured, acquired by the object image acquisition unit 12b, and the information indicating an image showing the background, acquired by the background image acquisition unit 13b. The feature quantity generation unit 14b generates an integrated feature quantity by integrating the feature quantity of the object, which is extracted based on the information indicating an image in which an object is captured, acquired by the object image acquisition unit 12b, and which incorporates the weight map generated by the weight map generation unit 15, with the feature quantity of the background, which is extracted based on the information indicating an image showing the background, acquired by the background image acquisition unit 13b. As a result, the feature extraction device 10b according to the second embodiment can extract feature quantities that enable even more accurate person identification than the first embodiment, in addition to the effects of the first embodiment.

[0147] Furthermore, the feature generation unit 14b includes an object feature extraction unit 141b that extracts object features, which are influenced by the weight map generated by the weight map generation unit 15, based on information acquired by the object image acquisition unit 12b and the weight map generated by the weight map generation unit 15; a background feature extraction unit 142b that extracts background features based on information acquired by the background image acquisition unit 13b; and a feature integration unit 143b that integrates the object features influenced by the weight map extracted by the object feature extraction unit 141b and the background features extracted by the background feature extraction unit 142b to generate integrated features. As a result, the feature extraction device 10b according to Embodiment 2 can integrate the object features influenced by the weight map and the background features to generate integrated features.

[0148] Furthermore, the object feature extraction unit 141b extracts object features by taking into account the weight map generated by the weight map generation unit 15 to the feature map used in the object feature extraction unit 141b. As a result, the feature extraction device 10b according to Embodiment 2 can extract object features with the weight map appropriately taken into account.

[0149] Furthermore, the object feature extraction unit 141b adjusts the size of the weight map generated by the weight map generation unit 15 to match the size of the feature map, and then incorporates the weight map into the feature map. As a result, the feature extraction device 10b according to Embodiment 2 can appropriately incorporate the weight map into the feature map, and can accurately extract the features of a person from which the influence of the background has been removed.

[0150] Furthermore, the object feature extraction unit 141b inputs the weight map generated by the weight map generation unit 15 to a feature extraction unit different from the object feature extraction unit 141b, extracts background difference features which are features of the weight map, and adds the extracted background difference features to the feature map used by the object feature extraction unit 141b, replacing the weight map generated by the weight map generation unit 15, before extracting the features of the object. As a result, the feature extraction device 10b according to Embodiment 2 can appropriately incorporate the background difference features extracted from the weight map into the feature map, and can extract the features of a person with even greater accuracy after the influence of the background has been removed.

[0151] Embodiment 3. Embodiment 1 described a feature extraction device 10 capable of extracting features that reduce the dependence on the accuracy of preprocessing compared to conventional methods, while suppressing the influence of background information on the identification accuracy of the object, and a learning device 70 comprising the feature extraction device 10. Embodiment 3 describes a learning device capable of generating a trained model for calculating the similarity between multiple integrated features, for example, a trained model in which the influence of background features is reduced.

[0152] Figure 12 shows an example of the configuration of a learning device 80 according to Embodiment 3. For example, as shown in Figure 12, the learning device 80 is configured to include a learning data storage unit 11' that includes a video information storage unit 11, a feature extraction device 10b that includes an object image acquisition unit 12b, a background image acquisition unit 13b, a feature quantity generation unit 14b, and a weight map generation unit 15, an intermediate feature loss calculation unit 81, a task loss calculation unit 82, a parameter update unit 83, and a trained model storage unit 84. In Figure 12, an example is shown in which the intermediate feature loss calculation unit 81 is included inside the feature extraction device 10b, but the intermediate feature loss calculation unit 81 is not limited to this and may be provided outside the feature extraction device 10b.

[0153] If the object feature extraction unit 141b is configured, for example, by a convolutional neural network, it outputs information (for example, a feature map) output from the intermediate layer of the convolutional neural network to the intermediate feature loss calculation unit 81.

[0154] If the background feature extraction unit 142b is composed of, for example, a convolutional neural network, it outputs information (for example, a feature map) output from the intermediate layer of the convolutional neural network to the intermediate feature loss calculation unit 81.

[0155] The weight map generation unit 15 generates a map (weight map) with different magnitudes of values ​​depending on the region, using either or both of the person image information acquired from the object image acquisition unit 12b and the background image information acquired from the background image acquisition unit 13b. The weight map generation unit 15 outputs the generated weight map to the intermediate feature loss calculation unit 81.

[0156] The intermediate feature loss calculation unit 81 acquires information output from the object feature extraction unit 141b, information output from the background feature extraction unit 142b, and the weight map output from the weight map generation unit 15. Based on the information acquired from the object feature extraction unit 141b and the background feature extraction unit 142b, and the weight map acquired from the weight map generation unit 15, the intermediate feature loss calculation unit 81 calculates a regularization term to be added to the loss function used by the task loss calculation unit 82. The intermediate feature loss calculation unit 81 outputs the calculated regularization term to the task loss calculation unit 82.

[0157] The regularization term described above acts to constrain parameter learning so that, for example, when extracting features from the background region common to both the person image and the background image, the object feature extraction unit 141b and the background feature extraction unit 142b extract similar features. For example, the intermediate feature loss calculation unit 81 assigns predetermined weights to the information obtained from the object feature extraction unit 141b and the background feature extraction unit 142b based on the weight map output from the weight map generation unit 15, calculates the mean squared error (MSE) between the weighted intermediate features, and uses the weighted sum of the MSEs between multiple intermediate features as the intermediate feature loss. This ensures that background regions common to both the person image and the background image are extracted as similar features. This makes it easier for the feature integration unit 143b to reduce the influence of background features when calculating integrated features.

[0158] The task loss calculation unit 82, similar to the task loss calculation unit 71 in Embodiment 1, obtains integrated features from the feature integration unit 143 and calculates a task loss for calculating the similarity between multiple integrated features based on the obtained integrated features and the annotation data stored in the training data storage unit 11'. At this time, the task loss calculation unit 82 calculates the task loss using a loss function to which a regularization term output from the intermediate feature loss calculation unit 81 has been added. The task loss calculation unit 82 outputs the calculated task loss to the parameter update unit 83.

[0159] The parameter update unit 83 obtains the task loss from the task loss calculation unit 82. Based on the task loss obtained from the task loss calculation unit 82, the parameter update unit 83 updates the parameters of the learning model. For example, the parameter update unit 83 updates the parameters of the learning model in a direction that minimizes the task loss obtained from the task loss calculation unit 82.

[0160] The calculation of the regularization term by the intermediate feature loss calculation unit 81, the calculation of the task loss by the task loss calculation unit 82, and the updating of the parameters of the learned model by the parameter update unit 83 may be performed multiple times. As a result, the parameter update unit 83 obtains a trained model that the object identification unit 53 uses to calculate the similarity between multiple integrated features. The parameter update unit 83 saves the obtained trained model in the trained model storage unit 84.

[0161] The trained model storage unit 84 is configured, for example, as an HDD or SSD. The trained model storage unit 84 stores the trained model obtained by the parameter update unit 83.

[0162] Figure 13 is a diagram illustrating an example of the operation of the learning device 80 according to Embodiment 3. In Figure 13, the area labeled "Background Feature Regularization Block" corresponds to the Intermediate Feature Loss Calculation Unit 81. Also in Figure 13, the area labeled "Backbone for People" corresponds to the Object Feature Extraction Unit 141b, and the area labeled "Backbone for Background" corresponds to the Background Feature Extraction Unit 142b.

[0163] As described above, the intermediate feature loss calculation unit 81 calculates a regularization term L_reg to be added to the loss function used by the task loss calculation unit 82, based on the information obtained from the object feature extraction unit 141b (such as pl1), the information obtained from the background feature extraction unit 142b (such as bl1), and the weight map obtained from the weight map generation unit 15. This regularization term L_reg acts to constrain the parameter learning so that, for example, when extracting features from the background region common to the person image and the background image, the object feature extraction unit 141b and the background feature extraction unit 142b extract similar features. Furthermore, as shown in Figure 13, the integrated features generated by the feature integration unit 143b pass through, for example, a fully connected layer and a softmax layer, and the task loss calculation unit 82 calculates the loss L_ID based on the features obtained after passing through the softmax layer and the ground truth data, using, for example, the loss function to which the regularization term L_reg has been added.

[0164] Thus, the learning device 80 is configured to include a learning data storage unit 11', a feature extraction device 10b including an object image acquisition unit 12b, a background image acquisition unit 13b, a feature generation unit 14b, and a weight map generation unit 15, a task loss calculation unit 82, a parameter update unit 83, and a trained model storage unit 84. This configuration allows the device to generate a trained model used by the object identification unit 53 to calculate the similarity between multiple integrated features, for example, a trained model in which the influence of background features is reduced. The trained model storage unit 84 may be located outside the learning device 80.

[0165] As described above, according to this embodiment 3, the learning device 80 includes a feature extraction device 10b that includes an object image acquisition unit 12b, a background image acquisition unit 13b, a feature generation unit 14b, and a weight map generation unit 15, and an intermediate feature loss calculation unit 81 that uses the weight map acquired from the weight map generation unit 15 to calculate a regularization term to be added to the loss function used by the task loss calculation unit 82. The task loss calculation unit 82 calculates the task loss using the loss function to which the regularization term calculated by the intermediate feature loss calculation unit 81 has been added. As a result, the learning device 80 according to embodiment 3 can generate a trained model for calculating the similarity between a plurality of integrated features, for example, a trained model in which the influence of background features is reduced.

[0166] Although preferred embodiments have been described in detail above, the invention is not limited to the embodiments described above, and various modifications and substitutions can be made to the embodiments described above without departing from the scope of the claims.

[0167] Furthermore, this disclosure allows for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0168] This disclosure enables the extraction of features that reduce the dependence on preprocessing accuracy compared to conventional methods, while suppressing the influence of background information on the identification accuracy of the object. It is suitable for use in feature extraction devices, object identification devices, learning devices, and feature extraction methods.

[0169] 1, 1b Object identification system, 2 Imaging device, 10, 10b Feature extraction device, 11 Video information storage unit, 11' Training data storage unit, 12, 12b Object image acquisition unit, 13, 13b Background image acquisition unit, 14, 14b Feature generation unit, 15 Weight map generation unit, 50 Object identification device, 51 Feature storage unit, 52 Feature acquisition unit, 53 Object identification unit, 61 Processing circuit, 62 CPU, 63 Memory, 70 Learning device, 71 Task loss calculation unit, 72 Parameter update unit, 73 Trained model storage unit, 80 Learning device, 81 Intermediate feature loss calculation unit, 82 Task loss calculation unit, 83 Parameter update unit, 84 Trained model storage unit, 100 Feature extraction device, 141, 141b Object feature extraction unit, 142, 142b Background feature extraction unit, 143, 143b Feature integration unit, 200 Object feature extraction unit, 500 Object identification device, 501 Feature storage unit, 502 Feature acquisition unit, 503 Object identification unit.

Claims

1. A feature extraction device comprising: an object image acquisition unit that acquires information indicating an image showing the object to be identified, obtained from information indicating a reference image showing the object to be identified and the background of the object, which is captured by an imaging device; a background image acquisition unit that acquires information indicating an image showing the background from information indicating an image captured by the imaging device; and a feature quantity generation unit that integrates the feature quantities of the object extracted based on the information acquired by the object image acquisition unit and the feature quantities of the background extracted based on the information acquired by the background image acquisition unit to generate integrated feature quantities for identifying the object.

2. The feature extraction apparatus according to claim 1, wherein the feature generation unit includes: an object feature extraction unit that extracts feature quantities of an object based on information acquired by the object image acquisition unit; a background feature extraction unit that extracts feature quantities of the background based on information acquired by the background image acquisition unit; and a feature integration unit that integrates the feature quantities of the object extracted by the object feature extraction unit and the feature quantities of the background extracted by the background feature extraction unit to generate integrated feature quantities.

3. The feature extraction apparatus according to claim 2, characterized in that the feature integration unit assigns a predetermined weight to at least one of the feature quantities of the object and the feature quantities of the background, and then integrates both set of feature quantities.

4. The feature extraction device according to any one of claims 1 to 3, wherein the background image acquisition unit acquires information indicating an image showing the background from information indicating a plurality of images taken by the imaging device, which are in a field of view corresponding to the field of view when the reference image was taken, and which were taken at a time different from the time when the reference image was taken.

5. The feature extraction device according to claim 4, characterized in that the background image acquisition unit obtains a plurality of pieces of information indicating images showing the background of the object based on each of the pieces of information indicating a plurality of images taken at a different time than the time the reference image was taken, and the feature generation unit individually integrates a plurality of background feature quantities extracted based on the plurality of pieces of information showing the background obtained by the background image acquisition unit with the feature quantities of the object extracted based on the information indicating the object, thereby generating a plurality of integrated feature quantities.

6. The feature extraction device according to claim 1, comprising a weight map generation unit that generates a weight map with different magnitudes of values ​​depending on the region, using either or both of the information indicating an image in which the object is captured, obtained by the object image acquisition unit, and the information indicating an image in which the background is captured, obtained by the background image acquisition unit, and integrating the feature quantities of the object extracted based on the information indicating an image in which the object is captured, obtained by the object image acquisition unit, and the feature quantities of the object that have been taken into account by the weight map generation unit, and the feature quantities of the background extracted based on the information indicating an image in which the background is captured, obtained by the background image acquisition unit, to generate the integrated feature quantities.

7. The feature extraction apparatus according to 6, wherein the feature generation unit includes: an object feature extraction unit that extracts the feature quantities of the object, which are the feature quantities of the object that take into account the weight map generated by the weight map generation unit, based on the information acquired by the object image acquisition unit and the weight map generated by the weight map generation unit; a background feature extraction unit that extracts the feature quantities of the background based on the information acquired by the background image acquisition unit; and a feature integration unit that integrates the feature quantities of the object that take into account the weight map, extracted by the object feature extraction unit, and the feature quantities of the background extracted by the background feature extraction unit, to generate the integrated feature quantities.

8. The feature extraction device according to claim 7, characterized in that the object feature extraction unit extracts the features of the object after taking into account the weight map generated by the weight map generation unit to the feature map used in the object feature extraction unit.

9. The feature extraction apparatus according to claim 8, characterized in that the object feature extraction unit adjusts the size of the weight map generated by the weight map generation unit to match the size of the feature map, and then adds the weight map to the feature map.

10. The feature extraction device according to claim 7, wherein the object feature extraction unit inputs the weight map generated by the weight map generation unit to a feature extraction unit different from the object feature extraction unit, extracts background difference features which are features of the weight map, and then, in place of the weight map generated by the weight map generation unit, adds the extracted background difference features to the feature map used by the object feature extraction unit, and then extracts the features of the object.

11. An object identification device comprising: a feature acquisition unit that acquires the integrated feature quantities generated by the feature extraction device described in any one of claims 1 to 10; and an object identification unit that identifies the object using the integrated feature quantities acquired by the feature acquisition unit.

12. A learning device comprising: a feature extraction device according to claim 1; a task loss calculation unit that calculates a task loss for calculating the similarity between a plurality of integrated features generated by the feature generation unit; and a parameter update unit that obtains a trained model for calculating the similarity by updating the parameters of the learning model based on the task loss calculated by the task loss calculation unit.

13. The learning device according to 12, comprising: a weight map generation unit that generates a weight map with different magnitudes of values ​​depending on the region, using either or both of the information indicating an image of the object obtained by the object image acquisition unit and the information indicating an image of the background obtained by the background image acquisition unit; and an intermediate feature loss calculation unit that calculates a regularization term to be added to the loss function used by the task loss calculation unit using the weight map generated by the weight map generation unit, wherein the task loss calculation unit calculates the task loss using the loss function to which the regularization term calculated by the intermediate feature loss calculation unit has been added.

14. A feature extraction method using a feature extraction device, comprising: a step of an object image acquisition unit acquiring information indicating an image showing the object, obtained from a reference image captured by an imaging device which shows the object to be identified and the background of the object; a step of a background image acquisition unit acquiring information indicating an image showing the background from information indicating an image captured by the imaging device; and a step of a feature quantity generation unit integrating the feature quantity of the object extracted based on the information acquired by the object image acquisition unit and the feature quantity of the background extracted based on the information acquired by the background image acquisition unit to generate an integrated feature quantity for identifying the object.

Images