Server device, system, server device control method, and storage medium

Abstract

Provided is a server device that facilitates identification of a person who has performed an act suspected of being a crime in an unmanned store or the like. The server device comprises an acquisition means, an authentication means, a first analysis means, and a control means. The acquisition means acquires: a certificate for certifying the identity and biological information of an authentication-target person; and biological information of the authentication-target person. The authentication means authenticates the authentication-target person by using the biological information obtained from the certificate and the biological information of the authentication-target person. When the authentication is successful, the authentication means permits the authentication-target person to enter a prescribed facility and stores the biological information and identity information on the identity of the authentication-target person. The first analysis means detects a prescribed action by an entry-permitted person by analyzing first image data obtained by capturing an image of the inside of the prescribed facility. When the prescribed action by the entry-permitted person is detected, the control means allows a person associated with the prescribed facility to refer to the identity information of the entry-permitted person who has performed the prescribed action and the first image data in which the entry-permitted person who has performed the prescribed action is shown.

Description

Server device, system, control method for server device, and storage medium

[0001] The present invention relates to a server device, a system, a control method for a server device, and a storage medium.

[0002] There are technologies related to crime prevention in unmanned stores.

[0003] For example, Patent Document 1 describes providing an unmanned store system and an unmanned store management method that can avoid replacement of users who have purchased products and appropriately identify users who have left the store without purchasing products in face authentication in a store. The system of Patent Document 1 includes an entrance checker (first face recognition machine), a cashier desk (second face recognition machine), and an exit checker (third face recognition machine). The entrance checker acquires a face image of a user at the time of entry using a first camera and performs processing related to face authentication for permitting the user to enter the store based on the face image at the time of entry. The cashier desk acquires a face image of the user at the time of settlement using a second camera and performs processing related to face authentication for permitting the user to settle based on the face image at the time of settlement. The exit checker acquires a face image of the user at the time of exit using a third camera and performs processing related to face authentication for confirming the user's exit based on the face image at the time of exit.

[0004] Japanese Unexamined Patent Application Publication No. 2020-166638

[0005] The system of Patent Document 1 grasps a person who has left the store without purchasing products by performing face authentication when a customer enters, settles, and exits the store. In the system of Patent Document 1, member registration is requested from users. Here, if the information registered at the time of member registration contains false information, it becomes difficult to identify the identity of a customer who has stolen products.

[0006] A main object of the present invention is to provide a server device, a system, a control method for a server device, and a storage medium that contribute to facilitating the identification of the identity of a person who has committed an act suspected of crime in an unmanned store or the like.

[0007] According to a first aspect of the present invention, a server device is provided comprising: an acquisition means for acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the predetermined facility using the stored biometric information of the person who has entered the predetermined facility; and a control means for making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the predetermined facility when a predetermined action by the person to be authenticated is detected.

[0008] According to a second aspect of the present invention, a system is provided comprising: an authentication terminal installed in a predetermined facility and a server device, wherein the server device includes: an acquisition means for acquiring from the authentication terminal an identification certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter the predetermined facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the identification certificate and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the identification certificate; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the predetermined facility using the stored biometric information of the person to be authenticated who has entered the predetermined facility; and a control means for making the identity information of the person to be authenticated who performed the predetermined action and the first image data showing the person to be authenticated who performed the predetermined action accessible to persons in charge of the predetermined facility when a predetermined action by the person to be authenticated is detected.

[0009] A third aspect of the present invention provides a control method for a server device, comprising: an acquisition step of acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated; an authentication step of authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis step of detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the predetermined facility using the stored biometric information of the person who entered the predetermined facility; and a control step of making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the predetermined facility when a predetermined action by the person to be authenticated is detected.

[0010] According to a fourth aspect of the present invention, a computer-readable storage medium is provided that stores a program for a computer mounted on a server device to execute: an acquisition process for acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated; an authentication process for authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis process for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the predetermined facility using the stored biometric information of the person who has entered the predetermined facility; and a control process for making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the predetermined facility when the predetermined action by the person to be authenticated is detected.

[0011] According to each aspect of the present invention, a server device, a system, a control method for the server device, and a storage medium are provided that contribute to facilitating the identification of a person who has committed a suspected crime in an unmanned store or the like. However, the effects of the present invention are not limited to those described above. The present invention may also produce other effects in lieu of or in conjunction with the effects described above.

[0012] Figure 1 is a diagram illustrating the outline of one embodiment. Figure 2 is a flowchart illustrating an example of the operation of one embodiment. Figure 3 is a diagram illustrating an example of the schematic configuration of an information processing system according to an embodiment of this disclosure. Figure 4 is a diagram illustrating an example of the configuration of an unmanned store according to an embodiment of this disclosure. Figure 5 is a diagram illustrating the operation of an information processing system according to an embodiment of this disclosure. Figure 6 is a diagram illustrating the operation of an information processing system according to an embodiment of this disclosure. Figure 7 is a diagram illustrating an example of the processing configuration of a server device according to an embodiment of this disclosure. Figure 8 is a diagram illustrating an example of a customer management database according to an embodiment of this disclosure. Figure 9 is a diagram illustrating an example of a person of interest management database according to an embodiment of this disclosure. Figure 10 is a flowchart illustrating an example of the operation of a person of interest control unit according to an embodiment of this disclosure. Figure 11 is a diagram illustrating the operation of a person of interest control unit according to an embodiment of this disclosure. Figure 12 is a flowchart illustrating an example of the operation of a monitoring control unit according to an embodiment of this disclosure. Figure 13 is a diagram illustrating the operation of a monitoring control unit according to an embodiment of this disclosure. Figures 14A and 14B are diagrams illustrating the operation of a suspicious person control unit according to an embodiment of this disclosure. Figure 15 is a diagram illustrating an example of the hardware configuration of a server device according to this disclosure.

[0013] First, an overview of one embodiment will be described. The reference numerals in the drawings attached to this overview are provided for convenience as examples to aid understanding, and this overview is not intended to be limiting in any way. Furthermore, unless otherwise specified, the blocks shown in each drawing represent functional units, not hardware units. The connecting lines between blocks in each drawing include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectional flow. In this specification and in the drawings, elements that can be similarly described are given the same reference numerals to avoid redundant explanation.

[0014] A server device 100 according to one embodiment includes an acquisition means 101, an authentication means 102, a first analysis means 103, and a control means 104 (see Figure 1). The acquisition means 101 acquires a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated (step S1 in Figure 2). The authentication means 102 authenticates the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated (step S2). If authentication is successful, the authentication means 102 permits the person to be authenticated to enter the predetermined facility and stores the identity information and biometric information of the person to be authenticated obtained from the certificate (step S3). The first analysis means 103 detects a predetermined action by the visitor by analyzing first image data obtained by photographing the inside of the predetermined facility using the stored biometric information of the visitor who has entered the predetermined facility (step S4). When the control means 104 detects a predetermined action by an entrant, it makes the identity information of the entrant who performed the predetermined action and a first image data showing the entrant who performed the predetermined action accessible to persons in charge of the predetermined facility (step S5).

[0015] The server device 100 acquires an identification document that guarantees the identity of the person to be authenticated, and authenticates the person attempting to enter a designated facility using the biometric information obtained from the identification document. At that time, the server device 100 acquires identity information from the identification document and stores it together with the biometric information. The server device 100 analyzes image data obtained from camera devices installed in the facility to detect a predetermined action (for example, a criminal act such as theft) performed by an entrant inside the facility. If such a predetermined action is detected, the server device 100 manages the identity information obtained from the identification document in association with the image data showing the predetermined action, enabling facility personnel to identify the entrant who performed the predetermined action after the fact. Through this operation of the server device 100, the identity of a person who has committed an act suspected of being criminal in an unmanned store or similar location can be easily identified.

[0016] Specific embodiments will be described in more detail below with reference to the drawings.

[0017] [First Embodiment] The first embodiment will be described in more detail with reference to the drawings.

[0018] [System Configuration] Figure 3 is a diagram showing an example of the schematic configuration of an information processing system according to the embodiment of the present disclosure. As shown in Figure 3, the information processing system includes a server device 10.

[0019] The server device 10 controls customer entry and exit in the unmanned store, as well as security for the unmanned store. The server device 10 may be installed in the building of the business operator running the unmanned store, or it may be installed on a network (on the cloud).

[0020] Customers visiting the unmanned store are carrying a device 20 such as a smartphone.

[0021] Figure 4 shows an example of the schematic configuration of an unmanned store. As shown in Figure 4, a door 30 is installed at the entrance of the unmanned store.

[0022] An authentication terminal 31 is installed on the outside of the unmanned store. The authentication terminal 31 is a device that serves as an interface for customers (persons being authenticated) to undergo authentication when entering the unmanned store. The authentication terminal 31 controls the opening and closing of the door 30 according to the authentication result.

[0023] The authentication terminal 31 includes a camera for photographing the person being authenticated, a communication device for connecting to the network, and a touch panel for displaying messages and accepting user input. The authentication terminal 31 also includes a device for short-range wireless communication with terminal 20, such as NFC (Near Field Communication).

[0024] Product shelves and at least one camera device 32 are installed in various locations within the unmanned store.

[0025] The server device 10, terminal 20, and each device installed in the unmanned store (authentication terminal 31, camera device 32) are connected to the network. Specifically, the server device 10, etc., are connected to the network by wired or wireless communication means.

[0026] The configurations of the information processing system and unmanned store shown in Figures 3 and 4 are illustrative examples and are not intended to limit the possible configurations. For example, the information processing system may include multiple server devices 10. Load balancing and redundancy may be achieved by using multiple server devices 10.

[0027] [Outline Operation] Next, the outline operation of the information processing system according to the first embodiment will be described.

[0028] <Preparing for a Digital Wallet> Users will use a digital wallet. A digital wallet is an electronic information storage service that guarantees information security, including data integrity, reliability, and availability.

[0029] By opening a digital wallet, users can store various digital content such as airline tickets, concert tickets, electronic money, and credit cards in that digital wallet.

[0030] <Obtaining Identity Verification Documents> Users who open a digital wallet acquire the digital content to be stored in that digital wallet. For example, in addition to the electronic money etc. mentioned above, users acquire "identity verification documents" that serve as proof of the issuer's identity and biometric information.

[0031] This section explains how users can obtain proof of identity from the municipality that issued their My Number Card.

[0032] Examples of biometric information include data (feature quantities) calculated from individual physical characteristics such as face, fingerprints, voiceprints, veins, retina, and iris patterns. Alternatively, biometric information may be image data such as face images or fingerprint images. Biometric information only needs to include information about the user's physical characteristics. This disclosure describes the case where biometric information related to a person's "face" (face image or feature quantities generated from a face image) is used.

[0033] First, terminal 20 reads biometric information (facial image) from the IC (Integrated Circuit) embedded in the My Number Card held by the user. Terminal 20 also acquires the user's biometric information (facial image) through a so-called selfie.

[0034] Terminal 20 performs identity verification using a facial image obtained from an IC chip and a selfie facial image. Terminal 20 performs identity verification by determining whether the two facial images belong to the same person.

[0035] If identity verification is successful, terminal 20 obtains a PIN (a four-digit number) from the user and uses the obtained PIN to retrieve a user authentication electronic certificate from the IC chip of the My Number Card. Terminal 20 then transmits the user authentication electronic certificate obtained from the IC chip to a local government server (not shown in Figure 3, etc.) managed by the local government that issued the My Number Card.

[0036] The local government server verifies the acquired user authentication electronic certificate. If the local government server successfully verifies the user authentication electronic certificate, it generates an identity verification document that verifies the identity (basic four pieces of information: name, gender, date of birth, address) and biometric information (facial image) of the My Number Card recipient corresponding to the user authentication electronic certificate. More precisely, the local government server generates an identity verification document that verifies the recipient's user ID, basic four pieces of information, and facial image.

[0037] Furthermore, the recipient's User ID may be a unique identifier for each recipient of a My Number Card (the electronic certificate obtained from the My Number Card). For example, the End User ID is used as an example of the User ID.

[0038] The local government server issues identity verification documents in the form of VCs (Verifiable Credentials), which allow for online verification of their contents. In the following explanation, identity verification documents issued as Credential Verification Documents (VCs) will be referred to as "Identity Verification Document VCs."

[0039] The local government server sends the generated identity verification certificates (VCs) to terminal 20. Terminal 20 stores the received identity verification certificates (VCs) in its digital wallet.

[0040] <Entering an unmanned store> To enter an unmanned store, the user (customer) moves in front of the authentication terminal 31 (see Figure 5).

[0041] When the authentication terminal 31 detects a user in front of it, it takes a picture of the user and acquires their biometric information.

[0042] Furthermore, the authentication terminal 31 instructs the user to submit their identity verification VCs. In response to this instruction, the user touches terminal 20 to the NFC reader of the authentication terminal 31. The authentication terminal 31 requests terminal 20 to submit their identity verification VCs. In accordance with this request, terminal 20 transmits the identity verification VCs stored in its digital wallet to the authentication terminal 31.

[0043] The authentication terminal 31 sends an "authentication request" to the server device 10, which includes the acquired biometric information (facial image) and identity verification documents (VCs).

[0044] The server device 10 verifies the personal identification certificate VCs. When the verification of the personal identification certificate VCs is successful, the server device 10 executes an authentication process using the biometric information obtained from the personal identification certificate VCs and the biometric information obtained by the authentication terminal 31 capturing the user (the person to be authenticated).

[0045] Specifically, the server device 10 executes one-to-one authentication using the two pieces of biometric information. The server device 10 transmits the authentication result (authentication success, authentication failure) to the authentication terminal 31.

[0046] Further, when the authentication of the person to be authenticated (the customer who intends to enter the unmanned store) is successful, the server device 10 stores the basic four pieces of information, user ID, and biometric information obtained from the personal identification certificate VCs in the store visitor management database. The details of the store visitor management database will be described later.

[0047] Upon receiving an authentication success, the authentication terminal 31 opens the door 30. The user passes through the door 30 and enters the unmanned store. After a predetermined time has elapsed since the door 30 was opened, the authentication terminal 31 closes the door 30.

[0048] <Detection of persons of concern> The camera device 32 transmits the image data obtained by capturing a predetermined area inside the unmanned store to the server device 10 at regular intervals or at a predetermined timing (see FIG. 6). More specifically, the camera device 32 transmits the image data and the camera ID to the server device 10.

[0049] The camera ID is an ID for identifying each camera device 32 set in the unmanned store. The MAC (Media Access Control) address or IP (Internet Protocol) address of the camera device 32 can be used as the camera ID.

[0050] The server device 10 accumulates the image data received for each camera device 32 (for each camera ID).

[0051] The server device 10 analyzes the accumulated image data (video data) and detects a person with a high likelihood of committing a crime such as shoplifting as a person of concern.

[0052] For example, the server device 10 analyzes the behavior of a customer using image data accumulated during a predetermined period of time (e.g., 5 minutes) after the customer enters the store. For example, the server device 10 uses the accumulated image data and a generated AI (Artificial Intelligence) model to calculate a "suspiciousness level" indicating the degree of suspicion (likelihood of committing a crime) regarding the customer's behavior.

[0053] For example, the server device 10 calculates a level of suspicion ranging from "0" to "1". A level of suspicion of "0" indicates that there is nothing suspicious about the customer's behavior, while a level of suspicion of "1" indicates that the customer's behavior is extremely suspicious.

[0054] The server device 10 performs threshold processing on the calculated suspiciousness level and treats customers entering the store with a suspiciousness level higher than a predetermined value as persons of concern. Specifically, the server device 10 manages persons of concern using a "caution flag." In doing so, the server device 10 assigns a caution flag level corresponding to the suspiciousness level to the person of concern.

[0055] For example, the server device 10 assigns a warning flag 1 (mild suspicion), a warning flag 2 (moderate suspicion), and a warning flag 3 (severe suspicion) to individuals of concern. In this way, the server device 10 assigns warning flags to individuals of concern, for example, categorized into three levels.

[0056] Server device 10 manages information on individuals with a warning flag assigned to them using a database of individuals with a warning flag. More specifically, server device 10 stores the user ID of an individual with a warning flag in the database of individuals with a warning flag. Details of the database of individuals with a warning flag will be described later.

[0057] Furthermore, the server device 10 determines the retention period for information on persons of concern (user ID, warning flag) according to the level of the warning flag. More specifically, the server device 10 sets a longer retention period for information on persons of concern as the level of the warning flag increases.

[0058] <Monitoring of Customer Behavior> The server device 10 monitors the behavior of each customer inside the store. More specifically, the server device 10 monitors whether customers are committing criminal acts such as theft (shoplifting). The server device 10 analyzes image data acquired from the camera device 32 to monitor the behavior of customers.

[0059] In this process, the server device 10 determines the frame rate of the image data used for analysis depending on whether or not a warning flag has been set for the person entering the store. More specifically, if the person entering the store does not have a warning flag set (not a person requiring attention), the server device 10 sets a lower frame rate during image data analysis. Conversely, if the person entering the store has a warning flag set (a person requiring attention), the server device 10 sets a higher frame rate during image data analysis.

[0060] Analyzing high-frame-rate image data increases the load on the server device 10. However, by using high-frame-rate image data for analysis, the server device 10 can more accurately determine whether or not criminal activity has occurred by customers. If the frame rate of the image data used for analysis is low, the moment of theft or other crime may not be captured. Taking this possibility into consideration, the server device 10 uses high-frame-rate image data for analyzing the behavior of highly suspicious individuals.

[0061] If, by analyzing the image data, a person highly likely to have committed a crime (hereinafter referred to as a suspicious person) is detected, the server device 10 performs an identification process for the suspicious person. The server device 10 stores the information obtained through the identification process for the suspicious person (identity information; basic four pieces of information) and the image data of the suspicious person so that the manager of the unmanned store can refer to it.

[0062] Next, we will describe the details of each device included in the information processing system according to the first embodiment.

[0063] Figure 7 shows an example of the processing configuration (processing module) of the server device 10 according to the embodiment disclosed herein. Referring to Figure 7, the server device 10 comprises a communication control unit 201, an authentication control unit 202, an image data control unit 203, a person of interest control unit 204, a monitoring control unit 205, a suspicious person control unit 206, and a storage unit 207.

[0064] The communication control unit 201 is a means for controlling communication with other devices. For example, the communication control unit 201 receives data (packets) from the terminal 20. The communication control unit 201 also transmits data to the terminal 20. The communication control unit 201 passes the data received from other devices to other processing modules. The communication control unit 201 transmits the data acquired from other processing modules to other devices. In this way, other processing modules send and receive data with other devices via the communication control unit 201. The communication control unit 201 has the function of a receiving unit that receives data from other devices and the function of a transmitting unit that transmits data to other devices.

[0065] The authentication control unit 202 is a means for performing control related to the authentication of a customer (person to be authenticated) who is attempting to enter an unmanned store.

[0066] The authentication control unit 202 has functions as both an acquisition means and an authentication means. The acquisition means acquires a certificate (identity certificate VCs) that certifies the identity and biometric information of a person to be authenticated who is attempting to enter a predetermined facility (unmanned store), and the biometric information of the person to be authenticated. The authentication means authenticates the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, permits the person to enter the predetermined facility. Furthermore, the authentication means stores identity information (e.g., basic four pieces of information) and biometric information related to the identity of the person to be authenticated obtained from the certificate (identity certificate VCs).

[0067] The authentication control unit 202 processes the authentication request received from the authentication terminal 31.

[0068] Upon receiving an authentication request, the authentication control unit 202 verifies the identity verification certificates (VCs) included in the authentication request. Specifically, the authentication control unit 202 verifies the electronic signature attached to the identity verification certificates (VCs), verifies that the identity verification certificates (VCs) have not been invalidated by the issuer, and verifies that the expiration date of the identity verification certificates (VCs) has not expired.

[0069] If the verification of the identity verification documents (VCs) fails, the authentication control unit 202 sets the authentication result to "Authentication Failed".

[0070] If the verification of the identity verification documents (VCs) is successful, the authentication control unit 202 performs one-to-one authentication (one-to-one matching) using the biometric information obtained from the identity verification documents (VCs) and the biometric information obtained by the authentication terminal 31 when it photographs the user (authenticated person).

[0071] Specifically, the authentication control unit 202 generates feature quantities from the facial image obtained from the identity verification document VCs (the facial image that the identity verification document VCs are intended to verify) and from the facial image obtained by the authentication terminal 31.

[0072] Regarding the feature generation process by the authentication control unit 202, existing technologies can be used, so a detailed explanation will be omitted. For example, the authentication control unit 202 extracts the eyes, nose, mouth, etc., from the face image as feature points. Then, the authentication control unit 202 calculates the position of each feature point and the distance between each feature point as feature quantities (generating a feature vector consisting of multiple feature quantities).

[0073] Once feature quantities are generated from two facial images, the authentication control unit 202 calculates the similarity between the two feature quantities. The chi-squared distance, Euclidean distance, and other similarity methods can be used to determine this similarity. Note that the greater the distance, the lower the similarity, and the closer the distance, the higher the similarity.

[0074] The authentication control unit 202 determines that authentication has failed if the calculated similarity score is less than a predetermined value. In this case, the authentication control unit 202 sets the authentication result to "Authentication Failed".

[0075] The authentication control unit 202 determines that authentication is successful if the calculated similarity score is above a predetermined level. In this case, the authentication control unit 202 sets the authentication result to "Authentication successful".

[0076] The authentication control unit 202 notifies the authentication terminal 31 of the authentication result (authentication successful, authentication failed). Specifically, if authentication is successful, the authentication control unit 202 sends an affirmative response to the authentication terminal 31. If authentication fails, the authentication control unit 202 sends a negative response to the authentication terminal 31.

[0077] Furthermore, upon successful authentication of the person to be authenticated, the authentication control unit 202 stores the date and time of entry, the four basic pieces of information obtained from the identity verification documents (VCs), the user ID, and biometric information (face image, features generated from the face image) in the visitor management database (see Figure 8).

[0078] The customer management database shown in Figure 8 is an example and is not intended to limit the items to be stored. Figure 8 also includes feature quantities generated from facial images as biometric information. The biometric information stored in the customer management database may be biometric information obtained from identity verification documents (VCs) or biometric information obtained by the authentication terminal 31 photographing the person being authenticated.

[0079] Once the information of the person entering the store is stored in the customer management database, the authentication control unit 202 notifies the person of concern control unit 204 of the user ID of the person entering the store.

[0080] The authentication control unit 202 deletes the entry (entry into the customer management database) of a customer leaving the unmanned store. Specifically, a camera is installed at the exit of the unmanned store. The authentication control unit 202 acquires the biometric information (facial image) of the customer leaving from the camera. The authentication control unit 202 performs a matching process (one-to-many matching; N is a positive integer) using the customer's biometric information and at least one other biometric information stored in the customer management database to identify the customer leaving. The authentication control unit 202 then deletes the entry of the identified customer leaving.

[0081] Through the operation of the authentication control unit 202, the customer management database stores the four basic pieces of information of customers staying in the unmanned store, as well as their user ID and biometric information (e.g., feature quantities).

[0082] In this manner, the authentication control unit 202 obtains a user ID that uniquely identifies the authenticated person (the person entering the unmanned store) from the terminal 20 held by the authenticated person via the authentication terminal 31. The authentication control unit 202 stores the ID, identity information (basic four pieces of information), and biometric information of the authenticated person who has been authorized to enter the designated facility in the visitor management database (first database).

[0083] The image data control unit 203 is a means for performing control over image data received from at least one camera device 32 installed in the unmanned store.

[0084] The image data control unit 203 stores the received image data for each camera device 32 (for each camera ID). The image data control unit 203 also deletes image data that has been stored for a predetermined time (for example, 1 hour).

[0085] The person under concern control unit 204 is a means for performing control related to persons under concern.

[0086] The person of concern control unit 204 functions as an analysis means (second analysis means). The person of concern control unit 204 calculates the degree of suspiciousness regarding the behavior of visitors to the unmanned store by analyzing the first image data. The person of concern control unit 204 sets visitors whose calculated degree of suspiciousness is equal to or greater than the first value as persons of concern. The person of concern control unit 204 assigns a warning flag of a level corresponding to the calculated degree of suspiciousness to visitors who have been set as persons of concern. The person of concern control unit 204 stores the user ID, warning flag, and the retention period of the user ID and warning flag, which are determined according to the level of the warning flag, in association with the person of concern management database (second database) for visitors who have been set as persons of concern.

[0087] As described above, information on customers identified as persons of concern is stored in the person of concern management database (see Figure 9). As shown in Figure 9, the person of concern management database stores the user ID, warning flag, and retention period of the person of concern. Note that the person of concern management database shown in Figure 9 is an example and is not intended to limit the items to be stored.

[0088] When the person under concern control unit 204 detects a person under concern by analyzing image data, it stores the user ID of the person under concern, the person's warning flag, and a retention period determined according to the level of the warning flag in the person under concern management database.

[0089] Furthermore, the person under concern control unit 204 accesses the person under concern management database periodically or at predetermined intervals and deletes entries whose retention period has expired.

[0090] Figure 10 is a flowchart showing an example of the operation of the person of interest control unit 204 according to the embodiment disclosed herein. The operation of the person of interest control unit 204 will be explained with reference to Figure 10.

[0091] When the authentication control unit 202 notifies the user ID of the person entering the store, the person of concern control unit 204 determines whether or not to perform a person of concern determination process for that person (determination of necessity of determination process; step S101).

[0092] Specifically, the person under suspicion control unit 204 searches the person under suspicion management database using the user ID obtained from the authentication control unit 202 as a key.

[0093] If the search is successful, the person under concern control unit 204 determines that it is unnecessary to perform a person under concern determination process for new customers entering the store. If the search is successful, it is determined that the customer has been previously determined to be a person under concern and therefore no re-determination is necessary. For example, if a person who has been determined to be a person under concern revisits the store before the retention period set in the person under concern management database expires, the search using the user ID as the key will be successful.

[0094] If the search fails, the Suspicious Person Control Unit 204 determines that it is necessary to perform a suspicious person determination process for new customers entering the store.

[0095] If the process for determining a person of concern is not required (step S102, No branch), the person of concern control unit 204 terminates the process.

[0096] If it is necessary to perform a process to determine if a person of concern is a person of concern (step S102, Yes branch), the person of concern control unit 204 performs the process to determine if a person of concern is a person of concern (step S103). Specifically, the person of concern control unit 204 analyzes the behavior of a new customer by analyzing the accumulated image data and determines whether or not the new customer is a person of concern.

[0097] For example, the person of concern control unit 204 acquires image data accumulated during a predetermined period (for example, 5 minutes) after the person of concern enters the store. The person of concern control unit 204 uses the image data accumulated during the predetermined period as the target of analysis for the person of concern determination process.

[0098] First, the person under suspicion control unit 204 extracts the area in which the person under suspicion is pictured from the acquired image data (at least one image data; video data).

[0099] For example, the person of interest control unit 204 extracts face regions from image data. Note that existing technologies can be used for the face image extraction process by the person of interest control unit 204, so a detailed explanation is omitted. For example, the person of interest control unit 204 may extract face images (face regions) from image data using a learning model trained by a CNN (Convolutional Neural Network). Alternatively, the person of interest control unit 204 may extract face images using methods such as template matching.

[0100] Once a facial region is extracted, the person under suspicion control unit 204 performs a matching process using the facial region (facial image) and the biometric information of the person to be judged (features stored in the customer management database) to determine whether the person in the image data is the person to be judged (a new customer). The person under suspicion control unit 204 performs a matching process (one-to-one matching) using the extracted facial image and the features stored in the customer management database to make the above determination.

[0101] If the person corresponding to the extracted facial image is not the person to be identified, the person under investigation control unit 204 does not perform any special processing.

[0102] If the person corresponding to the extracted facial image is a person to be identified, the person of concern control unit 204 processes the image data to indicate the area in which all or part of the person of concern is visible. For example, the person of concern control unit 204 outlines the area in which the entire body of the person of concern is visible with a solid line or the like (see Figure 11).

[0103] The person under suspicion control unit 204 repeatedly extracts the area in which the person under suspicion is captured and processes the data from the acquired image data (each image data constituting the video data). Furthermore, when image data obtained from multiple camera devices 32 is to be analyzed, the person under suspicion control unit 204 repeats the above processing for each image data stored separately for each camera ID.

[0104] When image data (video data) clearly showing the person to be judged is obtained, the person of interest control unit 204 inputs the image data into the generating AI model to obtain the suspiciousness level of the person to be judged. For example, the person of interest control unit 204 generates a prompt such as, "Please calculate the suspiciousness level of the actions of the person enclosed by a solid line in the image data, on a scale from 0 to 1," and inputs the image data along with the generated prompt into the generating AI.

[0105] The generating AI (for example, a large-scale language model) calculates a degree of suspiciousness based on the behavior of the person enclosed by the solid line. For example, a person who unnecessarily looks around or moves suspiciously (suspicious movement patterns) will be calculated as having a high degree of suspiciousness. The person of interest control unit 204 obtains the degree of suspiciousness from the generating AI.

[0106] The person of interest control unit 204 performs threshold processing on the acquired suspiciousness level and sets individuals whose suspiciousness level is higher than a predetermined value as persons of interest, and assigns a "caution flag" to them. At that time, the person of interest control unit 204 assigns a caution flag at a level corresponding to the suspiciousness level.

[0107] For example, if the level of suspicion is less than 0.5, the person under investigation control unit 204 will not classify the person as a person under investigation (will not assign a warning flag).

[0108] Conversely, if the level of suspicion is 0.5 or higher, the person under investigation control unit 204 determines that the person is a person under investigation (assigns a warning flag).

[0109] More specifically, for example, if the level of suspicion is 0.5 or higher but less than 0.75, the person of concern control unit 204 assigns a "warning flag 1" to the person being assessed, indicating that their behavior is mildly suspicious. If the level of suspicion is 0.75 or higher but less than 0.9, the person of concern control unit 204 assigns a "warning flag 2" to the person being assessed, indicating that their behavior is moderately suspicious. If the level of suspicion is 0.9 or higher, the person of concern control unit 204 assigns a "warning flag 3" to the person being assessed, indicating that their behavior is severely suspicious.

[0110] If the person being assessed is not a person of concern (step S104, No branch), the person of concern control unit 204 terminates the process.

[0111] If the person being assessed is a person of concern (step S104, Yes branch), the person of concern control unit 204 stores the information of the person who entered the store and was determined to be a person of concern in the person of concern management database (storing information of the person of concern; step S105). Specifically, as shown in Figure 9, the person of concern control unit 204 stores the user ID, warning flag, and retention period in the person of concern management database.

[0112] Furthermore, the retention period for information on individuals of concern (database entries; user ID, warning flag) is predetermined according to the warning flag level. More specifically, the retention period for information on individuals of concern (user ID and warning flag) increases as the warning flag level rises.

[0113] For example, if the warning flag is 1, the retention period is set to 1 day. Or, if the warning flag is 3, the retention period is set to 1 week. The person requiring attention control unit 204 calculates the retention period based on the current time and the retention period corresponding to the warning flag, and stores the said retention period in the person requiring attention management database.

[0114] The monitoring control unit 205 is a means for performing controls related to monitoring customers entering the store.

[0115] The monitoring control unit 205 functions as an analysis means (first analysis means). The monitoring control unit 205 analyzes first image data obtained by taking pictures inside a predetermined facility (unmanned store) using biometric information of visitors who have entered the facility stored in the visitor management database. By analyzing the first image data, the monitoring control unit 205 detects a predetermined action by a visitor (for example, shoplifting).

[0116] More specifically, the monitoring control unit 205 monitors whether customers entering the store are committing criminal acts such as theft. The monitoring control unit 205 analyzes the accumulated image data and monitors the behavior of customers entering the store. For example, the monitoring control unit 205 performs customer behavior monitoring processing periodically or at predetermined intervals.

[0117] For example, the monitoring control unit 205 performs behavior monitoring processing for each customer entering the store at 5-minute intervals.

[0118] Figure 12 is a flowchart showing an example of the operation of the monitoring control unit 205 according to the embodiment disclosed herein. The operation of the monitoring control unit 205 will be explained with reference to Figure 12.

[0119] First, the monitoring control unit 205 identifies the person appearing in the image data (step S201). Specifically, the monitoring control unit 205 acquires image data (video data) accumulated over a predetermined period (for example, 5 minutes) and identifies the person (enterer) appearing in each image data using biometric information stored in the entrant management database.

[0120] Specifically, the monitoring control unit 205 extracts the face region from the image data. The monitoring control unit 205 then performs a matching process using the extracted face region (face image) and biometric information (feature quantities) stored in the customer management database. More specifically, the monitoring control unit 205 sets the feature quantities generated from the extracted face image as the matching target and performs a one-to-many matching with multiple feature quantities registered in the customer management database.

[0121] The monitoring control unit 205 calculates the similarity between the feature quantity to be matched and each of the multiple feature quantities on the registered side. The monitoring control unit 205 identifies the person who entered the store as the person in the image data, corresponding to the entry with the feature quantity that has the highest similarity among the multiple feature quantities registered in the store visitor management database.

[0122] When a person is identified in the image data, the monitoring control unit 205 determines whether the identified person is a person of interest (determination of person of interest; step S202).

[0123] Specifically, the monitoring control unit 205 searches the database of persons under concern using the user ID of the identified person as the key. If the search fails, the monitoring control unit 205 determines that the identified person (enterer) is not a person under concern. If the search is successful, the monitoring control unit 205 determines that the identified person (enterer) is a person under concern.

[0124] When it is determined whether or not a person in the image data is a person of interest, the monitoring control unit 205 reflects the determination result in the image data (step S203).

[0125] For example, the monitoring control unit 205 outlines all or part of the area where non-suspicious individuals are captured with a solid line. Conversely, the monitoring control unit 205 outlines all or part of the area where suspicious individuals are captured with a dotted line. Furthermore, the monitoring control unit 205 writes the user ID of each individual near the frame.

[0126] The monitoring control unit 205 repeats the above processing for the acquired image data (each image data constituting the video data) and for each person captured in the image data. As a result, image data (video data) is obtained in which a user ID is set in the area in which each person entering the store is captured (see Figure 13).

[0127] Furthermore, when analyzing image data obtained from multiple camera devices 32, the monitoring and control unit 205 repeats the above processing for each image data stored separately for each camera ID.

[0128] Once image data reflecting the determination of whether each person entering the store is a person of interest is obtained, the monitoring control unit 205 generates image data for analyzing the behavior of the store entrants (generating data for behavioral analysis; step S204).

[0129] At that time, the monitoring control unit 205 generates behavioral analysis data for analyzing the behavior of non-suspicious individuals for whom a warning flag has not been set, and behavioral analysis data for analyzing the behavior of suspicious individuals for whom a warning flag has been set.

[0130] In the following explanation, behavioral analysis data used to analyze the behavior of non-suspicious individuals will be referred to as "behavioral analysis data (non-suspicious individuals)." Behavioral analysis data used to analyze the behavior of suspicious individuals will be referred to as "behavioral analysis data (suspicious individuals)."

[0131] The monitoring control unit 205 generates two sets of behavioral analysis data such that the frame rate of the behavioral analysis data (persons of interest) is higher than the frame rate of the behavioral analysis data (persons of interest).

[0132] For example, the monitoring control unit 205 generates low-frame-rate behavioral analysis data (non-suspicious individuals) by thinning out a large amount of data from image data in which each customer's user ID is set.

[0133] Alternatively, the monitoring and control unit 205 generates high-frame-rate behavioral analysis data (persons of interest) by thinning out the image data. Alternatively, the monitoring and control unit 205 may use the image data with each customer's user ID set as behavioral analysis data (persons of interest) without thinning out the data.

[0134] The monitoring control unit 205 uses the two types of behavioral analysis data generated to determine whether or not each customer has committed a crime (step S205).

[0135] For example, regarding customers who have not been assigned a warning flag (non-suspicious individuals), the monitoring control unit 205 generates a prompt such as, "Determine whether a criminal act has been committed by the person enclosed by a solid line in the image data. If a criminal act is detected, output the user ID of that person." The monitoring control unit 205 inputs the generated prompt and behavioral analysis data (non-suspicious individuals) into the generating AI.

[0136] Alternatively, for customers who have been flagged as "cautionary" (persons requiring attention), the monitoring control unit 205 generates a prompt such as, "Determine whether a criminal act has been committed by the person enclosed by the dotted line in the image data. If a criminal act is detected, output the user ID of that person." The monitoring control unit 205 inputs the generated prompt and behavioral analysis data (persons requiring attention) into the generating AI.

[0137] The monitoring and control unit 205 obtains the user ID of a store entrant from the generating AI if the entrant is committing a criminal act.

[0138] If no criminal activity is detected (step S206, No branch), the monitoring and control unit 205 terminates the process.

[0139] If criminal activity is detected (step S206, Yes branch), the monitoring control unit 205 treats the person suspected of committing the criminal activity as a suspicious person and outputs the user ID of the suspicious person to the suspicious person control unit 206 (step S207). Specifically, the monitoring control unit 205 passes the user ID obtained from the generated AI to the suspicious person control unit 206.

[0140] Thus, when the monitoring control unit 205 analyzes the behavior of customers who have been flagged with a warning, it uses first image data having a first frame rate. Conversely, when the monitoring control unit 205 analyzes the behavior of customers who have not been flagged with a warning, it uses first image data having a second frame rate. The first frame rate is set higher than the second frame rate.

[0141] The suspicious person control unit 206 is a means for performing controls on suspicious persons suspected of committing criminal acts. When a predetermined act by a store entrant is detected, the suspicious person control unit 206 controls the identification information of the store entrant who performed the predetermined act and a first image data showing the store entrant who performed the predetermined act so that it can be accessed by persons in charge of a predetermined facility (for example, the manager of an unmanned store).

[0142] When the monitoring control unit 205 obtains a user ID, the suspicious person control unit 206 searches the customer management database using the obtained user ID as a key and identifies the corresponding entry. The suspicious person control unit 206 then obtains four basic pieces of information (identity information) from the identified entry.

[0143] The suspicious person control unit 206 stores the acquired basic four pieces of information in association with image data (data for behavioral analysis) showing criminal acts committed by the suspicious person. This stored information (basic four pieces of information, image data) is referenced as needed by the store manager (person in charge) of the unmanned store.

[0144] For example, the manager of an unmanned store accesses the server device 10 using a personal computer or the like. The suspicious person control unit 206 provides the stored basic information 4 and image data in response to the request of the store manager. The store manager plays back the image data, and if a criminal act is found, reports the facts to the police or other relevant authorities.

[0145] As mentioned above, entries in the customer management database are deleted when a customer leaves the store. Therefore, the suspicious person control unit 206 needs to retrieve the four basic pieces of information about a customer from the customer management database when a criminal act by that customer is detected, and store this information so that the store manager or other personnel at the unmanned store can refer to it afterward.

[0146] The memory unit 207 is a means for storing information necessary for the operation of the server device 10.

[0147] [Terminal] Examples of terminals 20 include mobile devices such as smartphones, mobile phones, game consoles, and tablets. Terminal 20 can be any device or equipment as long as it can receive user input and communicate with the server device 10, etc. Furthermore, the configuration of terminal 20 is obvious to those skilled in the art, so a detailed explanation is omitted.

[0148] [Authentication Terminal] A detailed explanation of the configuration and operation of the authentication terminal 31 is omitted. The authentication terminal 31 obtains identity verification documents (VCs) from the authenticated person's terminal 20. The authentication terminal 31 also obtains the authenticated person's biometric information. The authentication terminal 31 obtains the authentication result (authentication successful, authentication failed) by transmitting the obtained identity verification documents (VCs) and biometric information to the server device 10. The authentication terminal 31 can then control the opening and closing of the door 30 according to the authentication result.

[0149] Next, a modified example of the first embodiment will be described.

[0150] <Modification 1> When the server device 10 detects a person of interest among the customers entering the store, it may notify a third-party committee of this fact. For example, the person of interest control unit 204 may notify a third-party committee, consisting of the manager of the unmanned store, that a person of interest has been detected. More specifically, the person of interest control unit 204 may send a message to the smartphone or other device carried by the manager of the unmanned store that a person of interest has been detected. Upon receiving this message, the manager of the unmanned store may go to the unmanned store or dispatch security guards to the unmanned store.

[0151] Alternatively, the person of interest control unit 204 may decide whether or not to notify the third-party committee that a person of interest has been detected, depending on the level (suspiciousness) of the warning flag assigned to the person of interest. For example, if a person of interest with warning flag 1 or warning flag 2 is detected, the person of interest control unit 204 will not notify the third-party committee. On the other hand, if a person of interest with warning flag 3 is detected, the person of interest control unit 204 will notify the third-party committee.

[0152] Thus, the person of concern control unit 204 may determine whether or not to report the detection of a person of concern to a third-party committee (escalate to a third-party committee) depending on the level of the warning flag.

[0153] <Modification 2> The server device 10 may store image data of persons of concern for a long period of time. More specifically, the person of concern control unit 204 may store information relating the user ID of a person of concern (a customer who has been flagged as a person of concern) to the image data of that person of concern.

[0154] For example, the person of interest control unit 204 associates user IDs with the storage locations of image data and stores them in the image storage database. In this case, it is desirable that the person of interest control unit 204 does not store the four basic pieces of information in the image storage database, taking into consideration the privacy of the person entering the store. The information stored in the image storage database will be referenced later if theft or other incidents are discovered and used as evidence to resolve the problem.

[0155] Furthermore, the information stored in the image storage database will be retained for a long period of time. Alternatively, the information stored in the image storage database may be deleted upon clear instructions from the manager of the unmanned store or other relevant personnel.

[0156] In this way, the person under suspicion control unit 204 associates the user ID of a store entrant who has been assigned a warning flag with information about the first image data in which the store entrant who has been assigned a warning flag is pictured, and stores this information in the image storage database (third database).

[0157] <Modification 3> The server device 10 may detect persons of interest based on the clothing, belongings, etc. of customers entering the store, instead of or in addition to the customer's actions after entering the store.

[0158] In this case, the authentication terminal 31 sends an authentication request to the server device 10 that includes image data showing the entire body of the person to be authenticated and identity verification documents (VCs). The server device 10 extracts biometric information (face image) from the image data and calculates the "blurriness" using the image data.

[0159] The degree of blurriness is an index that indicates the degree to which the hands of the person being authenticated are obscured by their clothing or belongings. For example, the range of blurriness is from "0" to "1". For example, blurriness "0" indicates that the hands of the person being authenticated are clearly visible, while blurriness "1" indicates that the hands of the person being authenticated are completely invisible.

[0160] More specifically, the authentication control unit 202 inputs the image data into the generating AI to obtain the degree of blurriness of the person being authenticated. The obtained blurriness, along with the user ID, is then passed on to the person of interest control unit 204.

[0161] The person of concern control unit 204 determines the level and retention period of the warning flag based on the acquired blurriness and stores it in the person of concern management database. For example, if the blurriness is less than 0.75, the person of concern control unit 204 treats the person being authenticated (the person wishing to enter the store) as not a person of concern. On the other hand, if the blurriness is 0.75 or higher, the person of concern control unit 204 treats the person being authenticated as a person of concern with warning flag 1.

[0162] In this way, the person of interest control unit 204 analyzes second image data showing the person to be authenticated attempting to enter a designated facility (unmanned store). By analyzing the second image data, the person of interest control unit 204 calculates the degree of blurriness regarding the hands of the person to be authenticated attempting to enter the designated facility. The person of interest control unit 204 sets any person (person to be authenticated attempting to enter the designated facility) whose calculated blurriness is equal to or greater than the second value as a person of interest.

[0163] <Modification 4> The server device 10 may refuse entry to a person whose user ID is registered as a suspicious person. Specifically, if the user ID obtained from the identity verification documents VCs is stored as the user ID of a suspicious person, the authentication control unit 202 may notify the authentication terminal 31 of the authentication failure.

[0164] Alternatively, if the user ID obtained from the identity verification documents (VCs) is stored in the suspected person management database as the user ID of a person of concern, the authentication control unit 202 may notify the authentication terminal 31 of the authentication failure. In particular, if the level of the warning flag for a person of concern is "3", the authentication control unit 202 may refuse entry to the person of concern.

[0165] <Modification 5> The server device 10 may apply cancelable encoding (encryption) to the user ID (end-user ID) obtained from the identity verification certificate VCs. In order to appropriately protect personal information, the server device 10 may store the encrypted user ID in the customer management database.

[0166] In this case, the person of interest control unit 204 may store the hash value obtained by inputting the hash key and user ID prepared for each unmanned store into a hash function in the customer management database. In this way, the person of interest control unit 204 may perform cancelable encoding on the ID unique to the identity verification document VCs.

[0167] Alternatively, terminal 20 may provide the authentication terminal 31 with an ID unique to the digital wallet (wallet ID; for example, wallet address) along with identity verification certificates (VCs). Server device 10 may use the wallet ID as a substitute for the user ID. By providing the wallet ID to server device 10, terminal 20 can also provide the authentication terminal 31 and server device 10 with an electronic student ID or electronic driver's license instead of identity verification certificates (VCs) issued by the local government. In other words, terminal 20 can provide the server device 10 with any certificate that verifies the identity and biometric information of the issuer as an identity verification document.

[0168] Alternatively, the server device 10 may generate a user ID using information obtained from the identity verification documents (VCs). For example, the server device 10 may generate a hash value from the concatenated value of name, gender, date of birth, and address obtained from the identity verification documents (VCs), and store the generated hash value as the user ID in the customer management database. The server device 10 can use any information that can uniquely identify a customer as the user ID.

[0169] <Modification 6> In the above embodiment, the server device 10 generated behavioral analysis data with a user ID set as shown in Figure 13, and the case where suspicious persons are detected (criminal activity is detected) using the generated behavioral analysis data was described. However, the server device 10 may also detect suspicious persons using behavioral analysis data (stored image data) without a user ID set.

[0170] Specifically, the monitoring control unit 205 inputs image data accumulated over a predetermined period (for example, 5 minutes) to the generating AI. At that time, the monitoring control unit 205 generates a prompt instructing the generating AI to output a facial image of a suspicious person suspected of committing a crime, and inputs this prompt to the generating AI.

[0171] The monitoring control unit 205 identifies the suspicious person's user ID by performing a matching process using the facial image output by the generating AI as the suspicious person's facial image and the biometric information stored in the customer management database. The monitoring control unit 205 then hands over the identified user ID to the suspicious person control unit 206.

[0172] <Modification 7> The server device 10 may analyze the image data obtained from the camera device 32 in real time to detect persons of interest or identify suspicious individuals.

[0173] For example, the Suspicious Person Control Unit 204 inputs the image data acquired by the Image Data Control Unit 203 into the AI ​​in real time, and instructs it to calculate the degree of suspicion by specifying the facial image of a new customer entering the store.

[0174] Alternatively, the monitoring control unit 205 inputs the image data acquired by the image data control unit 203 into the generating AI in real time and instructs it to output a facial image of a suspicious person who is highly likely to be committing a crime such as shoplifting.

[0175] <Modification 8> The monitoring control unit 205 may extract the section in which an act suspected of being a crime committed by a suspicious person is captured. For example, the monitoring control unit 205 may generate a prompt such as, "Please determine whether or not a criminal act has been committed by the person captured in the image data. If a criminal act is detected, please output the user ID of that person and the timestamp of the section in which the criminal act is captured." The monitoring control unit 205 may identify the section of video in which a criminal act by a suspicious person is captured by inputting the generated prompt into the generating AI.

[0176] The suspicious person control unit 206 may store image data showing the criminal act of the suspicious person in association with the four basic pieces of information of the suspicious person.

[0177] <Modification 9> The suspicious person control unit 206 may write the identity information (basic four pieces of information) of the suspicious person to the image data stored in association with the user ID of the suspicious person. For example, the suspicious person control unit 206 may generate image data (video data) as shown in Figure 14A.

[0178] Alternatively, the suspicious person control unit 206 may apply a mask to the faces of other shoppers who are pictured with the suspicious person. For example, as shown in Figure 14B, the suspicious person control unit 206 may black out the faces of people other than the suspicious person. Alternatively, the suspicious person control unit 206 may apply a mosaic effect to the faces of people other than the suspicious person.

[0179] The operation of the suspicious person control unit 206 provides image data that allows for easy viewing of video evidence of the crime committed by the suspicious person and the suspicious person's identity information. In addition, the masking process performed by the suspicious person control unit 206 appropriately protects the privacy of other customers entering the store.

[0180] <Modification 10> The monitoring control unit 205 may output image data (still image data) showing a clear criminal act by a suspicious person, along with the user ID of the suspicious person, to the suspicious person control unit 206. The suspicious person control unit 206 may acquire the four basic pieces of information of the suspicious person based on the acquired user ID, and store the acquired four basic pieces of information in association with the still image data, or write the four basic pieces of information to the still image data.

[0181] <Modification 11> In the above embodiment, the case in which the server device 10 detects a suspicious person from the actions of a store entrant was described. However, the final determination of whether or not a store entrant is a suspicious person may be made by a third-party committee consisting of the store manager, etc. (a determination committee whose members are the store manager, etc.).

[0182] In this case, the suspicious person control unit 206 obtains the user ID (the user ID of the store entrant that the generating AI determined to be a suspicious person) and the behavioral analysis data used by the generating AI to detect the suspicious person from the monitoring control unit 205. The suspicious person control unit 206 then transmits the acquired behavioral analysis data (image data that is presumed to show criminal activity by the store entrant) to the third-party committee.

[0183] The third-party committee will review the received video footage and determine whether or not a crime was committed by an entrant.

[0184] The third-party committee transmits the judgment result (criminal act present, no criminal act present) and authentication information to the server device 10. More specifically, the third-party committee uses a third-party committee terminal (not shown in Figure 3, etc.), such as a personal computer, to verify the image data and input the judgment result into the third-party committee terminal. The third-party committee also inputs each member's personal identification documents (VCs) or signed personal identification documents as authentication information into the third-party committee terminal. The third-party committee terminal transmits the judgment result and authentication information to the server device 10.

[0185] The intruder control unit 206 of the server device 10 verifies the received authentication information. If the authentication information is successfully verified, the intruder control unit 206 executes processing according to the determination result.

[0186] If the result indicates no criminal activity, the suspicious person control unit 206 discards the behavioral analysis data (image data).

[0187] If the system receives a result indicating criminal activity, the suspicious person control unit 206 searches the customer management database using the user ID as a key and identifies the corresponding entry. The suspicious person control unit 206 retrieves identity information (basic four pieces of information) from the identified entry. The suspicious person control unit 206 stores the retrieved identity information in association with the image data information.

[0188] Alternatively, the suspicious person control unit 206 may change the destination of the image data depending on the user ID of the suspicious person output by the generating AI and the likelihood (confidence level) of the result. For example, if the confidence level is low, the suspicious person control unit 206 sends the image data to a third-party committee. Conversely, if the confidence level is high (it is clear that the crime was committed by the person entering the store), the suspicious person control unit 206 sends the image data to an investigative agency such as the police.

[0189] Determining whether a customer (enterer) of an unmanned store is a suspicious person requires careful consideration. Therefore, the server device 10 entrusts the final decision on whether an entrant is a suspicious person to a third-party committee. If the third-party committee determines that criminal activity has occurred, the server device 10 identifies and stores the identity information of the suspicious person.

[0190] <Modification 12> The server device 10 may process the image data provided to the third-party committee to facilitate the members of the third-party committee's understanding of the circumstances at the time of the crime.

[0191] Specifically, the suspicious person control unit 206 may write the four basic pieces of information of the person entering the store, which the generating AI has identified as a suspicious person, into the image data. Furthermore, the suspicious person control unit 206 may apply a mask to the face area of ​​each person, including the suspicious person, that appears in the image data.

[0192] As described above, the server device 10 according to the first embodiment acquires the identity verification documents (VCs) of a person attempting to enter an unmanned store, and authenticates the person using the biometric information obtained from the identity verification documents (VCs). Upon successful authentication of the person, the server device 10 stores the identity information obtained from the identity verification documents (VCs) and the biometric information of the person entering the store in the store entrant management database. The server device 10 detects criminal acts such as shoplifting by store entrants by analyzing image data obtained from camera devices 32 installed in the unmanned store. If a criminal act by a store entrant is detected, the server device 10 manages the identity information obtained from the identity verification documents (VCs) in association with the image data showing the criminal act, enabling facility personnel (for example, the manager of the unmanned store) to identify the suspicious person afterward. Through this operation of the server device 10, the identity of a person suspected of committing a crime in an unmanned store can be easily and reliably identified. Furthermore, the operation of such a server device 10 reduces the human and time costs involved in identifying and tracking suspicious individuals.

[0193] Furthermore, the operation of this information processing system enables reliable tracking of individuals who steal goods while minimizing physical changes to the unmanned store. In addition, with the information processing system disclosed in this application, users of the unmanned store do not need to register as members in advance. Users can enter the unmanned store by touching a smartphone or other device containing their identity verification documents (VCs) to the authentication terminal 31. Thus, the information processing system disclosed in this application does not require users to register in advance for each store, and obtains identity verification documents (VCs) that guarantee the user's identity information through authentication when the user enters the unmanned store. Therefore, it does not impose a significant burden on users of the unmanned store. In other words, with the information processing system, store users do not need to register in advance for each store, and the authentication that store users receive is limited to once upon entry, and there are no restrictions on the user's actions after entering the store. Therefore, the information processing system disclosed in this application can improve store security without compromising the convenience of store users.

[0194] Furthermore, the information processing system disclosed in this application does not employ any methods to prevent individuals suspected of shoplifting or similar activities from leaving the store. Therefore, there is little concern that such individuals may damage the equipment of the unmanned store.

[0195] Next, we will describe the hardware of each device that makes up the information processing system. Figure 15 shows an example of the hardware configuration of the server device 10.

[0196] The server device 10 can be configured as an information processing device (a so-called computer), and has the configuration illustrated in Figure 15. For example, the server device 10 includes a processor 311, memory 312, input / output interface 313, and communication interface 314, etc. The components of the processor 311, etc. are connected by an internal bus or the like and are configured to communicate with each other.

[0197] However, the configuration shown in Figure 15 is not intended to limit the hardware configuration of the server device 10. The server device 10 may include hardware not shown, and it may not have to include the input / output interface 313 if necessary. Also, the number of processors 311 etc. included in the server device 10 is not intended to be limited to the example in Figure 15; for example, the server device 10 may include multiple processors 311.

[0198] The processor 311 is a programmable device such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or DSP (Digital Signal Processor). Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs, including an operating system (OS).

[0199] Memory 312 can be RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Memory 312 stores the OS program, application programs, and various data.

[0200] The input / output interface 313 is an interface for a display device or input device (not shown). The display device is, for example, a liquid crystal display. The input device is, for example, a device that accepts user input such as a keyboard or mouse.

[0201] The communication interface 314 is a circuit, module, etc., that communicates with other devices. For example, the communication interface 314 may include a NIC (Network Interface Card), etc.

[0202] The functions of the server device 10 are realized by various processing modules. These processing modules are realized, for example, by the processor 311 executing a program stored in the memory 312. The program can also be recorded on a computer-readable storage medium. The storage medium can be a non-transitory material such as semiconductor memory, hard disk, magnetic recording medium, or optical recording medium. In other words, the present invention can also be embodied as a computer program product. Furthermore, the program can be downloaded via a network or updated using the storage medium on which the program is stored. Moreover, the processing module may be realized by a semiconductor chip.

[0203] Furthermore, authentication terminals 31 and terminals 20, etc., can also be configured using information processing devices, similar to the server device 10. Since their basic hardware configuration is no different from that of the server device 10, a detailed explanation will be omitted.

[0204] The server device 10, which is an information processing device, is equipped with a computer, and its functions can be realized by having the computer execute a program. Furthermore, the server device 10 executes a control method for the server device 10 using this program.

[0205] [Modification] Note that the configuration and operation of the information processing system described in the above embodiment are illustrative examples and are not intended to limit the system configuration.

[0206] In the above embodiment, the configuration and operation of the information processing system were explained using an unmanned store as an example. However, the information processing system may also perform entry and exit control for other facilities, etc. For example, the information processing system may perform entry and exit control for a manned store or identify suspicious persons.

[0207] In the above embodiment, the case in which the terminal 20 performs identity verification using a My Number Card when requesting the issuance of identity verification certificates (VCs) from the local government was described. However, this identity verification may be omitted. The terminal 20 may also request the issuance of identity verification certificates (VCs) from the local government when the user enters the correct PIN.

[0208] Terminal 20 may request the issuance of identity verification certificates (VCs) from an entity other than the issuer of the My Number Card. For example, terminal 20 may request the issuance of identity verification certificates (VCs) from the Ministry of Foreign Affairs, which issues passports. In this case, terminal 20 may send the passport number read from the IC chip of the passport to a server managed by the Ministry of Foreign Affairs and request the issuance of identity verification certificates (VCs).

[0209] Terminal 20 may provide server device 10 with facial recognition certificates (VCs) that verify biometric information (e.g., facial information) instead of personal identification certificates (VCs) that verify identity and biometric information. In this case, server device 10 may allow the person to enter the store if it succeeds in authentication using the facial image obtained from the facial recognition certificates and the facial image obtained by photographing the person to be authenticated. Server device 10 may also analyze the image data of the person entering the store and estimate the person's gender and age. Server device 10 may treat the estimated gender, age, etc., as the identity information described above.

[0210] Terminal 20 may generate an identity certificate using information read from an identification document such as a My Number Card. For example, terminal 20 may generate an identity certificate by attaching an electronic signature to the four basic pieces of information and biometric information obtained from the IC chip of the My Number Card.

[0211] In the above embodiment, the case in which the authentication terminal 31 obtains identity verification certificates (VCs) from the terminal 20 using an NFC reader was described. However, the authentication terminal 31 may obtain identity verification certificates (VCs) using other means. For example, the authentication terminal 31 and the terminal 20 may communicate with each other using Bluetooth® to send and receive identity verification certificates (VCs). Alternatively, the terminal 20 may convert the identity verification certificates (VCs) into a two-dimensional code format in response to user operation and display the two-dimensional code. The authentication terminal 31 may decode the two-dimensional code displayed on the terminal 20 and obtain the identity verification certificates (VCs).

[0212] The server device 10 may calculate the degree of suspicion of a store entrant when monitoring their behavior (when a suspicious person is detected). The monitoring control unit 205 may perform the process of detecting suspicious persons and calculating the degree of suspicion in parallel using the accumulated image data. If the monitoring control unit 205 determines that a store entrant is a person of interest based on the calculated degree of suspicion, it may store the information of that store entrant in the person of interest management database.

[0213] The server device 10 may store the level of suspicion and warning flag for each customer in the customer management database. The monitoring control unit 205 does not need to perform customer behavior monitoring (suspicious person detection processing) if it accesses the customer management database and finds that no person of concern is among the customers. Alternatively, the monitoring control unit 205 may reduce the frequency of execution of the suspicious person detection processing, or it may not need to perform the suspicious person detection processing using high frame rate image data.

[0214] The server device 10 may calculate the degree of suspiciousness or detect suspicious behavior of shoppers (suspicious persons) without using generated AI. For example, the server device 10 may calculate the movement path (trajectory) of shoppers from image data and calculate the degree of suspiciousness based on that movement path. Alternatively, the server device 10 may detect suspicious behavior by shoppers using sensors installed in the store.

[0215] The server device 10 may generate high-frame-rate image data and low-frame-rate image data when accumulating image data. The image data control unit 203 may pre-generate and store behavioral analysis data (non-suspicious individuals) and behavioral analysis data (suspicious individuals) for use by the suspicious person control unit 206 for analysis.

[0216] The server device 10 may change the frame rate of the image data used for image analysis according to the degree of suspicion of each customer entering the store. Alternatively, the server device 10 may change the execution frequency of the suspicious person detection process according to the degree of suspicion of the customer and the level of the warning flag.

[0217] The server device 10 may set a period for storing information (user ID) of a person of concern according to the level of the warning flag assigned to that person. For example, the server device 10 may set a shorter retention period for user IDs of persons of concern with a low warning flag level, and a longer retention period for user IDs of persons of concern with a high warning flag level.

[0218] In the above embodiment, the operation of the server device 10 was explained using the example of a case where feature quantities are stored as biometric information in the customer management database. However, facial images may also be stored as biometric information in the customer management database. In this case, the server device 10 only needs to generate feature quantities from the facial images stored in the customer management database each time it uses a facial image stored in the customer management database.

[0219] In the above embodiment, we described a case where the customer management database and the person under concern management database are configured inside the server device 10, but these databases may be built on an external database server or the like. In other words, some functions of the server device 10 may be implemented on another server. More specifically, it is sufficient that the "monitoring control unit (monitoring and control means)" etc. described above are implemented in any device included in the system.

[0220] The form of data transmission and reception between each device (server device 10, authentication terminal 31) is not particularly limited, but the data transmitted and received between these devices may be encrypted. Since user biometric information and other data are transmitted and received between these devices, it is desirable that encrypted data be transmitted and received in order to properly protect this information.

[0221] In the flowcharts (sequence diagrams) used in the above description, multiple processes are shown in order, but the execution order of the processes performed in the embodiment is not limited to the order in which they are shown. In the embodiment, the order of the illustrated processes can be changed to the extent that it does not impede the content, for example, by executing each process in parallel.

[0222] The embodiments described above are explained in detail to facilitate understanding of the disclosure, and it is not intended that all the configurations described above are necessary. Furthermore, when multiple embodiments are described, each embodiment may be used individually or in combination. For example, it is possible to replace parts of the configuration of one embodiment with those of another embodiment, or to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of one embodiment with those of another.

[0223] As described above, the industrial applicability of the present invention is clear, and it is particularly suitable for use in information processing systems that perform access control and security control at a given facility.

[0224] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0225] [Note 1] A server device comprising: an acquisition means for acquiring an identification card that certifies the identity and biometric information of a person to be authenticated who intends to enter a designated facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the identification card and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the identification card; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the person to be authenticated who has entered the designated facility; and a control means for making the identity information of the person to be authenticated who performed the predetermined action and the first image data showing the person to be authenticated who performed the predetermined action accessible to persons in charge of the designated facility when a predetermined action by the person to be authenticated is detected.

[0226] [Note 2] The server device as described in Note 1, wherein the acquisition means acquires an ID that uniquely identifies the authenticated person, and the authentication means stores the ID, identity information, and biometric information of the authenticated person who has been permitted to enter the designated facility in association with each other in the first database.

[0227] [Appendix 3] The server device according to Appendix 2, further comprising: a second analysis means that analyzes the first image data to calculate a degree of suspicion regarding the visitor's behavior; designates visitors whose calculated degree of suspicion is equal to or greater than the first value as persons of concern; assigns a warning flag of a level corresponding to the calculated degree of suspicion to the visitors designated as persons of concern; and stores in a second database the ID of the visitor designated as a person of concern, the warning flag, and the retention period of the ID and the warning flag, which is determined according to the level of the warning flag, in association with each other.

[0228] [Note 4] The server device according to Note 3, wherein the second analysis means analyzes second image data showing a person to be authenticated who is attempting to enter the predetermined facility to calculate the degree of blurriness regarding the hands of the person to be authenticated who is attempting to enter the predetermined facility, and sets the person to be authenticated who is attempting to enter the predetermined facility if the calculated degree of blurriness is equal to or greater than the second value.

[0229] [Note 5] The server device described in Note 4, wherein the first analysis means uses the first image data having a first frame rate when analyzing the behavior of visitors who have been assigned the warning flag, and uses the first image data having a second frame rate when analyzing the behavior of visitors who have not been assigned the warning flag, and the first frame rate is higher than the second frame rate.

[0230] [Note 6] The server device described in Note 5, wherein the first analysis means stores in a third database information relating the ID of an attendee to whom the warning flag has been assigned and the first image data of the attendee to whom the warning flag has been assigned.

[0231] [Note 7] The acquisition means is a server device according to any one of Notes 1 to 6 that acquires the certificate issued in the form of VCs (Verifiable Credentials).

[0232] [Note 8] A system comprising: an authentication terminal installed in a designated facility; and a server device, wherein the server device includes: an acquisition means for acquiring from the authentication terminal an identification certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter the designated facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the identification certificate and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the identification certificate; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the person to be authenticated who has entered the designated facility; and a control means for making the identity information of the person to be authenticated who performed the predetermined action and the first image data showing the person to be authenticated who performed the predetermined action accessible to persons in charge of the designated facility when a predetermined action by the person to be authenticated is detected.

[0233] [Note 9] The system as described in Note 8, wherein the acquisition means acquires an ID that uniquely identifies the authenticated person, and the authentication means stores the ID, identity information, and biometric information of the authenticated person who has been permitted to enter the designated facility in association with each other in the first database.

[0234] [Appendix 10] The system according to Appendix 9, further comprising: a second analysis means that analyzes the first image data to calculate the degree of suspicion regarding the visitor's behavior; designates visitors whose calculated degree of suspicion is equal to or greater than the first value as persons of concern; assigns a warning flag of a level corresponding to the calculated degree of suspicion to the visitors designated as persons of concern; and stores in a second database the ID of the visitor designated as a person of concern, the warning flag, and the retention period of the ID and the warning flag, which is determined according to the level of the warning flag, in association with each other.

[0235] [Note 11] The system according to Note 10, wherein the second analysis means analyzes second image data showing a person to be authenticated who is attempting to enter the predetermined facility to calculate the degree of blurriness regarding the hands of the person to be authenticated who is attempting to enter the predetermined facility, and sets the person to be authenticated who is attempting to enter the predetermined facility if the calculated degree of blurriness is equal to or greater than the second value.

[0236] [Note 12] The system as described in Note 11, wherein the first analysis means uses the first image data having a first frame rate when analyzing the behavior of visitors who have been assigned the warning flag, and uses the first image data having a second frame rate when analyzing the behavior of visitors who have not been assigned the warning flag, and the first frame rate is higher than the second frame rate.

[0237] [Note 13] The system described in Note 12, wherein the first analysis means stores in a third database information relating the ID of the visitor to whom the warning flag has been assigned and the first image data of the visitor to whom the warning flag has been assigned.

[0238] [Note 14] The acquisition means is a system described in any one of Notes 8 to 13 that acquires the certificate issued in the form of VCs (Verifiable Credentials).

[0239] [Note 15] A control method for a server device comprising: an acquisition step of acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a designated facility, and the biometric information of the person to be authenticated; an authentication step of authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis step of detecting a predetermined action by the visitor by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the visitor who has entered the designated facility; and a control step of making the identity information of the visitor who performed the predetermined action and the first image data showing the visitor who performed the predetermined action accessible to persons in charge of the designated facility when a predetermined action by the visitor is detected.

[0240] [Note 16] The server device control method described in Note 15, wherein the acquisition step acquires an ID that uniquely identifies the person to be authenticated, and the authentication step stores the ID, identity information, and biometric information of the person to be authenticated who has been permitted to enter the designated facility in association with each other in a first database.

[0241] [Appendix 17] A control method for a server device as described in Appendix 16, further comprising a second analysis step of: analyzing the first image data to calculate a degree of suspicion regarding the visitor's behavior; setting visitors whose calculated degree of suspicion is equal to or greater than the first value as persons of concern; assigning a warning flag of a level corresponding to the calculated degree of suspicion to the visitors designated as persons of concern; and storing in a second database the ID of the visitor designated as a person of concern, the warning flag, and the retention period of the ID and the warning flag, which is determined according to the level of the warning flag, in association with each other.

[0242] [Note 18] The control method for the server device described in Note 17, wherein the second analysis step involves analyzing second image data showing a person to be authenticated who is attempting to enter the predetermined facility, calculating the degree of blurriness regarding the hands of the person to be authenticated who is attempting to enter the predetermined facility, and setting a person to be authenticated who is attempting to enter the predetermined facility whose calculated blurriness is equal to or greater than the second value as a person of concern.

[0243] [Note 19] The control method for the server device described in Note 18, wherein the first analysis step uses the first image data having a first frame rate when analyzing the behavior of visitors who have been assigned the warning flag, and uses the first image data having a second frame rate when analyzing the behavior of visitors who have not been assigned the warning flag, and the first frame rate is higher than the second frame rate.

[0244] [Note 20] The control method for the server device described in Note 19, wherein the first analysis step involves associating the ID of the visitor to whom the warning flag has been assigned with information regarding the first image data in which the visitor to whom the warning flag has been assigned is stored in a third database.

[0245] [Note 21] The acquisition step is the control method for a server device according to any one of Notes 15 to 20, wherein the acquisition step is to acquire the certificate issued in the form of VCs (Verifiable Credentials).

[0246] [Note 22] A computer-readable storage medium that stores a program for causing a computer mounted on a server device to execute: an acquisition process for acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated; an authentication process for authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis process for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by taking a photograph of the inside of the predetermined facility using the stored biometric information of the person who has entered the predetermined facility; and a control process for making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the predetermined facility if a predetermined action by the person to be authenticated is detected.

[0247] [Note 23] The storage medium described in Note 22, wherein the acquisition process acquires an ID that uniquely identifies the authenticated person, and the authentication process stores the ID, identity information, and biometric information of the authenticated person who has been permitted to enter the designated facility in association with each other in the first database.

[0248] [Note 24] The storage medium described in Note 23, further comprising: analyzing the first image data to calculate the degree of suspiciousness regarding the visitor's behavior; designating visitors whose calculated degree of suspiciousness is equal to or greater than the first value as persons of interest; assigning a warning flag of a level corresponding to the calculated degree of suspiciousness to the visitors designated as persons of interest; and storing in a second database the ID of the visitors designated as persons of interest, the warning flag, and the retention period of the ID and the warning flag, which is determined according to the level of the warning flag, in association with each other.

[0249] [Note 25] The storage medium described in Note 24, wherein the second analysis process analyzes second image data showing a person to be authenticated who is attempting to enter the predetermined facility to calculate the degree of blurriness regarding the hands of the person to be authenticated who is attempting to enter the predetermined facility, and sets a person to be authenticated who is attempting to enter the predetermined facility whose calculated blurriness is equal to or greater than the second value.

[0250] [Note 26] The first analysis process uses the first image data having a first frame rate when analyzing the behavior of visitors who have been assigned the warning flag, and uses the first image data having a second frame rate when analyzing the behavior of visitors who have not been assigned the warning flag, wherein the first frame rate is higher than the second frame rate, as described in Note 25.

[0251] [Note 27] The storage medium described in Note 26, wherein the first analysis process associates the ID of the visitor to whom the warning flag has been assigned with the first image data of the visitor to whom the warning flag has been assigned and stores this information in a third database.

[0252] [Note 28] The acquisition process is a storage medium according to any one of Notes 22 to 27, which acquires the certificate issued in the form of VCs (Verifiable Credentials).

[0253] Furthermore, some or all of the configurations described in Appendices 2 to 7, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 8, 15, and 22 in the same way as those described in Appendices 2 to 7. Moreover, not limited to Appendices 1 and 7, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0254] Furthermore, each disclosure of the above-mentioned prior art documents cited herein is incorporated herein by reference. Although embodiments of the present invention have been described above, the present invention is not limited to these embodiments. It will be understood by those skilled in the art that these embodiments are merely illustrative and that various modifications are possible without departing from the scope and spirit of the present invention. That is, the present invention naturally includes the entire disclosure, including the claims, and various modifications and alterations that can be made by those skilled in the art in accordance with the technical idea.

[0255] 10 Server device 20 Terminal 30 Door 31 Authentication terminal 32 Camera device 100 Server device 101 Acquisition means 102 Authentication means 103 First analysis means 104 Control means 201 Communication control unit 202 Authentication control unit 203 Image data control unit 204 Person of interest control unit 205 Monitoring control unit 206 Suspicious person control unit 207 Storage unit 311 Processor 312 Memory 313 Input / output interface 314 Communication interface

Claims

1. A server device comprising: an acquisition means for acquiring an identification card that certifies the identity and biometric information of a person to be authenticated who intends to enter a designated facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the identification card and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the identification card; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the person to be authenticated who has entered the designated facility; and a control means for making the identity information of the person to be authenticated who performed the predetermined action and the first image data showing the person to be authenticated who performed the predetermined action accessible to persons in charge of the designated facility when a predetermined action by the person to be authenticated is detected.

2. The server device according to claim 1, wherein the acquisition means acquires an ID that uniquely identifies the authenticated person, and the authentication means stores the ID, identity information, and biometric information of the authenticated person who has been permitted to enter the predetermined facility in association with each other in a first database.

3. The server device according to claim 2, further comprising: a second analysis means for: calculating a degree of suspicion regarding the behavior of the visitor by analyzing the first image data; designating visitors whose calculated degree of suspicion is equal to or greater than the first value as persons of concern; assigning a warning flag of a level corresponding to the calculated degree of suspicion to the visitors designated as persons of concern; and storing in a second database the ID of the visitor designated as a person of concern, the warning flag, and the retention period of the ID and the warning flag, which is determined according to the level of the warning flag, in association with each other.

4. The server device according to claim 3, wherein the second analysis means analyzes second image data showing a person to be authenticated who is attempting to enter the predetermined facility to calculate the degree of blurriness regarding the hands of the person to be authenticated who is attempting to enter the predetermined facility, and sets a person to be authenticated who is attempting to enter the predetermined facility whose calculated blurriness is equal to or greater than the second value as a person of concern.

5. The server device according to claim 4, wherein the first analysis means uses first image data having a first frame rate when analyzing the behavior of visitors to whom the warning flag has been assigned, and uses first image data having a second frame rate when analyzing the behavior of visitors to whom the warning flag has not been assigned, and the first frame rate is higher than the second frame rate.

6. The server device according to claim 5, wherein the first analysis means stores in a third database information relating the ID of an attendee to whom the warning flag has been assigned and the first image data of the attendee to whom the warning flag has been assigned.

7. The server device according to any one of claims 1 to 6, wherein the acquisition means acquires the certificate issued in the form of VCs (Verifiable Credentials).

8. A system comprising: an authentication terminal installed in a designated facility; and a server device, wherein the server device includes: an acquisition means for acquiring from the authentication terminal an identification certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter the designated facility, and the biometric information of the person to be authenticated; an authentication means for authenticating the person to be authenticated using the biometric information obtained from the identification certificate and the biometric information of the person to be authenticated, and if authentication is successful, permitting the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the identification certificate; a first analysis means for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the person to be authenticated who has entered the designated facility; and a control means for making the identity information of the person to be authenticated who performed the predetermined action and the first image data showing the person to be authenticated who performed the predetermined action accessible to persons in charge of the designated facility when a predetermined action by the person to be authenticated is detected.

9. A control method for a server device, comprising: an acquisition step of acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a designated facility, and the biometric information of the person to be authenticated; an authentication step of authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the designated facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis step of detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by photographing the inside of the designated facility using the stored biometric information of the person who entered the designated facility; and a control step of making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the designated facility, if a predetermined action by the person to be authenticated is detected.

10. A computer-readable storage medium that stores a program for causing a computer mounted on a server device to execute: an acquisition process for acquiring a certificate that certifies the identity and biometric information of a person to be authenticated who intends to enter a predetermined facility, and the biometric information of the person to be authenticated; an authentication process for authenticating the person to be authenticated using the biometric information obtained from the certificate and the biometric information of the person to be authenticated, and if authentication is successful, allowing the person to enter the predetermined facility, and storing the identity information and biometric information of the person to be authenticated obtained from the certificate; a first analysis process for detecting a predetermined action by the person to be authenticated by analyzing first image data obtained by taking a photograph of the inside of the predetermined facility using the stored biometric information of the person who has entered the predetermined facility; and a control process for making the identity information of the person who performed the predetermined action and the first image data showing the person who performed the predetermined action accessible to persons in charge of the predetermined facility if a predetermined action by the person to be authenticated is detected.

Images