Authentication system and method

Abstract

An onboard system according to the present invention comprises: an imaging device that images the face of a user present outside an automobile; and a transmission unit that transmits, to a management device, a captured image of the face of the user as captured by the imaging device. The management device comprises: a facial authentication unit that executes a facial authentication process that uses the captured image; a number-of-times determination unit that, on the basis of the result of the facial authentication process, determines whether the number of times that facial authentication has failed is equal to or greater than a threshold value; and a transmission unit that, when it has been determined that the number of times that facial authentication failed is equal to or greater than the threshold value, transmits a reading proposal instruction regarding identification information of the user to a portable terminal device and / or the onboard system. The portable terminal device comprises a proposal unit for proposing reading of the identification information of the user when the reading proposal instruction is received.

Description

Authentication System and Method

[0001] The present disclosure relates to an authentication system and method for authenticating a user of a vehicle.

[0002] Conventionally, various techniques for authenticating a user who intends to board a vehicle by face authentication have been proposed. As an example of such a technique, in the in-vehicle device control system disclosed in Patent Document 1, an authentication processing unit collates biometric information acquired by a camera with biometric information of a person who is permitted to unlock a vehicle door stored in a storage unit, and calculates a similarity between the pieces of biometric information. When the calculated similarity is smaller than a preset value, the door lock control device does not unlock the vehicle door.

[0003] Japanese Unexamined Patent Application Publication No. 2007-126144

[0004] However, in the in-vehicle device control system disclosed in Patent Document 1, when face authentication based on a captured image of the user's face fails, the user cannot grasp the fact, which causes a problem of low convenience.

[0005] One object of the present disclosure is to provide an authentication system and method capable of improving user convenience when authenticating a vehicle user in view of the above problem.

[0006] The authentication system according to the present disclosure includes: an in-vehicle system installed in a vehicle; a management device; and a mobile terminal device. The in-vehicle system includes: an imaging device that captures an image of a face of a user located outside the vehicle; and a transmitting unit that transmits the captured image of the user's face captured by the imaging device to the management device. The management device includes: a face authentication unit that executes face authentication processing using the captured image; a number-of-times determination unit that determines whether the number of failed face authentication attempts is equal to or greater than a threshold based on a result of the face authentication processing; and a transmitting unit that transmits a reading proposal instruction for the user's identification information when it is determined that the number of failed face authentication attempts is equal to or greater than the threshold. The mobile terminal device further includes a proposing unit that proposes reading of the user's identification information when receiving the reading proposal instruction.

[0007] The method described herein involves an in-vehicle system capturing a user's face while they are outside the vehicle, the in-vehicle system transmitting the captured image of the user's face to a management device, the management device performing facial recognition processing using the captured image, the management device determining whether the number of failed facial recognition attempts exceeds a threshold based on the results of the facial recognition processing, the management device sending a read proposal instruction for the user's identification information if it determines that the number of failed facial recognition attempts exceeds a threshold, and the user's mobile terminal device proposing to read the user's identification information upon receiving the read proposal instruction.

[0008] This disclosure provides an authentication system and method that can improve user convenience when authenticating vehicle users.

[0009] This is a schematic diagram showing an example of the authentication system related to this disclosure. This is a diagram showing an example of the hardware configuration of the in-vehicle system related to this disclosure. This is a diagram showing an example of the hardware configuration of the ECU (Electronic Control Unit) included in the in-vehicle system related to this disclosure. This is a diagram showing an example of the hardware configuration of the management device related to this disclosure. This is a diagram showing an example of the hardware configuration of the mobile terminal device related to this disclosure. This is a diagram showing an example of the functional configuration of the ECU related to this disclosure. This is a diagram showing an example of the functional configuration of the management device related to this disclosure. This is a diagram showing an example of the identification information table. This is a diagram showing an example of the functional configuration of the mobile terminal device related to this disclosure. This is a sequence diagram showing an example of the processing performed in the authentication system. This is a sequence diagram showing another example of the processing performed in the authentication system. This is a diagram showing an example of the processing performed by the management device. This is a diagram showing another example of the processing performed by the management device. This is a diagram showing the main components of the authentication system related to this disclosure. This is a diagram showing the main processing performed in the authentication system related to this disclosure.

[0010] Hereinafter, an exemplary embodiment will be described with reference to the drawings. Figure 1 is a schematic diagram showing an example of an authentication system 1 according to the present disclosure. The authentication system 1 includes an in-vehicle system 11 installed in a vehicle 10, a management device 20, and a portable terminal device 30. The management device 20 communicates data with the in-vehicle system 11 and the portable terminal device 30 via a network 40. The in-vehicle system 11 and the portable terminal device 30 perform wireless communication. The network 40 may be configured to include various wide-area communication networks. Although Figure 1 shows one vehicle 10, the authentication system 1 may include multiple vehicles 10 and multiple portable terminal devices 30.

[0011] Figure 2 shows an example of the hardware configuration of the in-vehicle system 11 according to this disclosure. The in-vehicle system 11 comprises an ECU 12, a camera 13, a speaker 14, and a communication interface (I / F) 15. The ECU 12 exchanges data with the camera 13, speaker 14, and communication interface 15 via a communication cable 16.

[0012] The ECU 12 controls the in-vehicle system 11. Details of the configuration and functions of the ECU 12 will be described later.

[0013] Camera 13 is a camera that captures images of the exterior of the automobile 10. Camera 13 is positioned to capture the face of a person attempting to get into the automobile 10.

[0014] Speaker 14 outputs a predetermined sound under the control of ECU 12. Speaker 14 corresponds to the sound output unit. Communication interface (I / F) 15 is an interface that performs data communication with other devices via an antenna (not shown).

[0015] Figure 3 shows an example of the hardware configuration of an ECU 12 included in the in-vehicle system 11 according to this disclosure. The ECU 12 comprises a processor 120, a memory 121, a storage device 122, a communication interface (I / F) 123, and a bus 124. These devices exchange data via the bus 124.

[0016] The processor 120 is a device that performs overall control of the ECU 12. Specific examples of the processor 120 include processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), and MPU (Micro Processing Unit). The processor 120 loads instructions contained in the program stored in the storage device 122 into the memory 121 and executes them. In other embodiments, at least a portion of the functions performed by the processor 120 may also be performed by integrated circuits such as FPGAs (Field-Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits). These devices are equivalent to computers. Furthermore, the ECU 12 can be equipped with multiple processors 120.

[0017] The storage device 122 is a storage device that stores various data, such as programs executed by the processor 120 and data processed by the processor 120. The communication interface 123 is an interface for data communication between the ECU 12 and other devices.

[0018] Figure 4 shows an example of the hardware configuration of the management device 20 according to this disclosure. The management device 20 comprises a processor 21, memory 22, storage device 23, communication interface (I / F) 24, and bus 25. These devices exchange data via the bus 25.

[0019] The processor 21 is a device that performs overall control of the management device 20. Specific examples of the processor 21 include CPUs, GPUs, MPUs, and other processors. The processor 21 loads instructions contained in the program stored in the storage device 23 into the memory 22 and executes them. In other embodiments, at least a portion of the functions performed by the processor 21 may also be performed by integrated circuits such as FPGAs and ASICs. These devices are equivalent to computers. Furthermore, the management device 20 may be equipped with multiple processors 21.

[0020] The storage device 23 is a storage device that stores various data, such as programs executed by the processor 21 and data processed by the processor 21. The communication interface 24 is an interface for data communication between the management device 20 and other devices.

[0021] Figure 5 shows an example of the hardware configuration of a mobile terminal device 30 according to this disclosure. The mobile terminal device 30 comprises a processor 31, a memory 32, a storage device 33, a communication interface (I / F) 34, an IC (Integrated Circuit) chip reader 35, a display 36, and a bus 37. These devices exchange data via the bus 37.

[0022] The processor 31 performs overall control of the mobile terminal device 30. Specific examples of the processor 31 include CPUs, GPUs, MPUs, and other processors. The processor 31 loads instructions contained in the program stored in the storage device 33 into the memory 32 and executes them. In other embodiments, at least a portion of the functions performed by the processor 31 may also be performed by integrated circuits such as FPGAs and ASICs. These devices are equivalent to computers. Furthermore, the mobile terminal device 30 can be equipped with multiple processors 31.

[0023] The storage device 33 is a storage device that stores various data, such as programs executed by the processor 31 and data processed by the processor 31. The communication interface 34 is an interface for data communication between the mobile terminal device 30 and other devices.

[0024] The IC chip reader 35 reads the IC chip on which the user's identification information (user ID) is recorded and obtains the user ID. Specific examples of IC chips include those embedded in identification cards such as driver's licenses, My Number cards, passports, employee IDs, and health insurance cards, as well as cash cards and credit cards.

[0025] The display 36 displays various images under the control of the processor 31. The display 36 corresponds to the proposal unit.

[0026] Figure 6 shows an example of the functional configuration of the ECU 12 according to this disclosure. The ECU 12 includes a receiving unit 110, an image capture control unit 111, a sound output control unit 112, a lock release unit 113, and a transmission unit 114. These functions can be implemented by a program.

[0027] The receiving unit 110 receives various information transmitted by other devices. The shooting control unit 111 causes the camera 13 to perform shooting processing to generate an image of the user's face. The shooting control unit 111 may be configured to use a human presence sensor to detect when a person approaches the camera 13 and cause the camera 13 to perform shooting processing.

[0028] The sound output control unit 112 outputs various sounds from the speaker 14. The lock release unit 113 unlocks the doors of the automobile 10. The transmission unit 114 transmits various information to other devices.

[0029] Figure 7 shows an example of the functional configuration of the management device 20 according to this disclosure. The management device 20 includes a receiving unit 200, a facial recognition unit 201, an authentication result determination unit 202, a count recording unit 203, a count determination unit 204, an identification information authentication unit 205, and a transmission unit 206. These functions can be implemented by a program.

[0030] The receiving unit 200 receives various information transmitted by other devices.

[0031] The facial recognition unit 201 performs facial recognition processing using the captured image provided by the in-vehicle system 11 and the user's face image. The facial recognition unit 201 can perform facial recognition processing using a facial recognition model, which is a machine learning model. When the facial recognition model receives a pair of images showing a person's face, it calculates a score indicating the probability that the people in these images are the same person. The facial recognition model is trained to output a high score when it receives a pair of images showing the faces of the same person. Conversely, the facial recognition model is trained to output a low score when it receives a pair of images showing the faces of different people. The facial recognition model may also be trained during the operation of the authentication system 1.

[0032] The authentication result determination unit 202 determines whether or not facial recognition by the facial recognition process was successful. Specifically, the authentication result determination unit 202 determines that facial recognition was successful if the score output by the facial recognition model is equal to or greater than a predetermined threshold. On the other hand, if the score output by the facial recognition model is less than the predetermined threshold, the authentication result determination unit 202 determines that facial recognition failed. The predetermined threshold can be any value.

[0033] Furthermore, the authentication result determination unit 202 determines whether or not the authentication of the user ID by the identification information authentication process was successful, based on the authentication result of the identification information authentication process performed by the identification information authentication unit 205.

[0034] The count recording unit 203 counts the number of failed facial recognition attempts and stores information indicating the number of failed facial recognition attempts in the storage device 23.

[0035] The count determination unit 204 determines whether the number of failed facial recognition attempts is equal to or greater than a predetermined threshold. The predetermined threshold can be any value. This threshold may also be set by the user.

[0036] The identification information authentication unit 205 performs an identification information authentication process to determine whether the user ID provided by the mobile terminal device 30 matches any of the user IDs associated with the user's identification information (mobile terminal ID) of the mobile terminal device 30 provided by the mobile terminal device 30, and outputs the authentication result. If these user IDs match, the identification information authentication unit 205 outputs a result indicating that the user ID authentication was successful. On the other hand, if these user IDs do not match, the identification information authentication unit 205 outputs a result indicating that the user ID authentication failed.

[0037] The identification information authentication unit 205 can refer to the identification information table shown in Figure 8 and identify the user ID associated with the mobile terminal ID provided by the mobile terminal device 30. The identification information table registers the vehicle ID, the identification information of the user's facial image (image ID), the identification information of the user's mobile terminal device 30 (mobile terminal ID), and at least one user ID in association with each other. The vehicle ID is the identification information of the vehicle used by the user identified by the user ID. The image ID is the identification information of the user's facial image identified by the user ID. The mobile terminal ID is the identification information of the user's mobile terminal device 30 identified by the user ID. A user may have multiple user IDs. These user IDs are user IDs recorded on each IC chip owned by the user.

[0038] The transmitting unit 206 transmits various types of information to other devices.

[0039] Figure 9 shows an example of the functional configuration of a mobile terminal device 30 according to this disclosure. The mobile terminal device 30 includes a receiving unit 300, a display control unit 301, a reading control unit 302, and a transmitting unit 303. These functions can be implemented by a program.

[0040] The receiving unit 300 receives various information transmitted by other devices. The display control unit 301 displays various images on the display 36. The reading control unit 302 reads the user ID recorded on the IC chip using the IC chip reader 35. The transmitting unit 303 transmits various information to other devices.

[0041] Figure 10 is a sequence diagram showing an example of processing executed in the authentication system 1. In step S1, an imaging control unit 111 of an in-vehicle system 11 causes a camera 13 to execute imaging processing, and generates a captured image of a user's face. In step S2, a transmission unit 114 of the in-vehicle system 11 transmits the generated captured image of the user's face and a vehicle ID of the vehicle in which the in-vehicle system 11 is installed to a management device 20.

[0042] When the management device 20 receives the captured image and the vehicle ID transmitted by the in-vehicle system 11, in step S3, a face authentication unit 201 of the management device 20 executes face authentication processing using the received captured image and a user's face image associated with the received vehicle ID. The face authentication unit 201 can refer to an identification information table to specify the user's face image associated with the received vehicle ID. In step S4, an authentication result determination unit 202 of the management device 20 determines whether face authentication through the face authentication processing succeeded or failed. An example where face authentication fails is described below.

[0043] If face authentication fails, in step S5, a count determination unit 204 of the management device 20 determines whether or not the number of failed face authentication attempts is equal to or greater than a predetermined threshold. An example where the number of failed face authentication attempts is less than the threshold is described below.

[0044] When it is determined that the number of failed face authentication attempts is less than the threshold, in step S6, a transmission unit 206 of the management device 20 transmits an authentication result indicating that face authentication failed to the in-vehicle system 11. In step S7, the transmission unit 206 of the management device 20 transmits the authentication result indicating that face authentication failed to a mobile terminal device 30.

[0045] When the in-vehicle system 11 receives the authentication result indicating that face authentication failed, in step S8, a sound output control unit 112 of the in-vehicle system 11 outputs a voice indicating that face authentication has failed from a speaker 14.

[0046] When the mobile terminal device 30 receives the authentication result indicating that face authentication failed, in step S9, a display control unit 301 of the mobile terminal device 30 displays an image indicating that face authentication has failed on a display 36.

[0047] FIG. 11 is a sequence diagram showing another example of processing executed in the authentication system 1. Steps S1 to S5 are the same as steps S1 to S5 in FIG. 10. Hereinafter, an example where the number of face authentication failures in step S5 is equal to or greater than a threshold will be described.

[0048] When it is determined that the number of face authentication failures is equal to or greater than the threshold, in step S10, the transmission unit 206 of the management device 20 transmits a reading proposal instruction for identification information to the in-vehicle system 11. In step S11, the transmission unit 206 of the management device 20 transmits a reading proposal instruction for identification information to the mobile terminal device 30.

[0049] When the in-vehicle system 11 receives the reading proposal instruction, in step S12, the sound output control unit 112 of the in-vehicle system 11 outputs, from the speaker 14, a sound suggesting that reading of a user ID be performed.

[0050] When the mobile terminal device 30 receives the reading proposal instruction, in step S13, the display control unit 301 of the mobile terminal device 30 displays, on the display 36, an image suggesting that reading of a user ID be performed. In step S14, the user causes the mobile terminal device 30 to read an IC chip on which the user's own user ID is recorded, whereby the reading control unit 302 of the mobile terminal device 30 reads the user ID recorded on the IC chip via the IC chip reader 35. In step S15, the transmission unit 303 of the mobile terminal device 30 transmits the read user ID and the mobile terminal ID of the mobile terminal device 30 to the management device 20.

[0051] When the management device 20 receives the user ID and the mobile terminal ID transmitted by the mobile terminal device 30, in step S16, the identification information authentication unit 205 of the management device 20 executes an identification information authentication process and outputs an authentication result of the user ID. In step S17, the authentication result determination unit 202 of the management device 20 determines whether authentication of the user ID succeeds or fails based on the authentication result of the user ID. Here, when authentication of the user ID succeeds, in step S18, the transmission unit 206 of the management device 20 transmits an unlock instruction to the in-vehicle system 11.

[0052] When the in-vehicle system 11 receives a lock release instruction, in step S19, the lock release unit 113 of the in-vehicle system 11 releases the door lock of the automobile 10.

[0053] Figure 12 shows an example of a process performed by the management device 20. The process shown in Figure 12 is performed when the management device 20 receives the captured image and vehicle ID transmitted by the in-vehicle system 11.

[0054] In step S20, the facial recognition unit 201 of the management device 20 performs facial recognition processing using the received captured image and vehicle ID. In step S21, the authentication result determination unit 202 determines whether or not facial recognition by the facial recognition processing was successful. If it is determined that facial recognition was successful (YES), in step S22, the transmission unit 206 of the management device 20 sends an unlock instruction to the in-vehicle system 11, and the process shown in Figure 12 ends.

[0055] On the other hand, if it is determined that facial recognition has failed (NO), the process branches to step S23. In step S23, the count determination unit 204 determines whether the number of facial recognition failures is above a predetermined threshold. If it is determined that the number of facial recognition failures is above a predetermined threshold (YES), in step S24, the transmission unit 206 transmits a reading suggestion instruction to the in-vehicle system 11 and the mobile terminal device 30, and the process shown in Figure 12 ends.

[0056] On the other hand, if it is determined that the number of facial recognition failures is less than a predetermined threshold (NO), in step S25, the count recording unit 203 saves information indicating the number of facial recognition failures to the storage device 23, and the process shown in Figure 12 ends.

[0057] Figure 13 shows another example of a process performed by the management device 20. The process in Figure 13 is performed when the management device 20 receives the user ID and mobile terminal ID transmitted by the mobile terminal device 30.

[0058] In step S30, the identification information authentication unit 205 of the management device 20 performs identification information authentication processing using the received user ID and mobile terminal ID, and outputs the authentication result of the user ID. In step S31, the authentication result determination unit 202 determines whether or not the authentication of the user ID was successful based on the authentication result of the user ID. If it is determined that the authentication of the user ID was successful (YES), in step S32, the transmission unit 206 of the management device 20 sends an unlock instruction to the in-vehicle system 11, and the process shown in Figure 13 ends.

[0059] On the other hand, if it is determined that user ID authentication has failed (NO), the process branches to step S33. In step S33, the transmission unit 206 sends a reading suggestion instruction for the user ID of another identification document to the in-vehicle system 11 and the mobile terminal device 30, and the process shown in Figure 13 ends.

[0060] When the in-vehicle system 11 receives a reading suggestion instruction for the user ID of another identification document, the sound output control unit 112 of the in-vehicle system 11 outputs an audio message from the speaker 14 indicating that it is suggesting the reading of the user ID contained in the other identification document. Meanwhile, when the mobile terminal device 30 receives a reading suggestion instruction for the user ID contained in the other identification document, the display control unit 301 of the mobile terminal device 30 displays an image on the display 36 indicating that it is suggesting the reading of the user ID contained in the other identification document.

[0061] Figure 14 is a diagram showing the main components of the authentication system 1 according to this disclosure. Figure 15 is a diagram showing the main processes performed in the authentication system 1 according to this disclosure. The camera 13 of the in-vehicle system 11 takes a photograph of the face of a user who is outside the automobile 10 (step S40). The transmission unit 114 of the in-vehicle system 11 transmits the captured image of the user's face to the management device 20 (step S41).

[0062] The facial recognition unit 201 of the management device 20 performs facial recognition processing using the captured image (step S42). The count determination unit 204 of the management device 20 determines, based on the results of the facial recognition processing, whether the number of times facial recognition has failed is above a threshold (step S43). If the transmission unit 206 of the management device 20 determines that the number of times facial recognition has failed is above a threshold, it transmits a user ID reading suggestion instruction (step S44).

[0063] When the mobile terminal device 30 receives a reading suggestion instruction, the suggestion unit 36 ​​of the mobile terminal device 30 suggests reading the user ID (step S45). The mobile terminal device 30 can suggest reading the user ID by displaying an image or outputting an audio message indicating that it is suggesting reading the user ID.

[0064] By adopting this configuration, users can be informed via voice when facial recognition fails and when their identification information is being read. Furthermore, users can confirm the failure of facial recognition and the reading of their user ID on the screen of their mobile device 30. Therefore, user convenience can be improved.

[0065] Furthermore, in the above-described embodiment, when the reading unit 35 of the mobile terminal device 30 receives a reading suggestion instruction, it reads the user ID contained in the user's identification document. The transmitting unit 303 of the mobile terminal device 30 transmits the read user ID to the management device 20. The identification information authentication unit 205 of the management device 20 performs authentication processing using the user ID.

[0066] If authentication of the user ID fails, the transmission unit 206 of the management device 20 sends a reading suggestion instruction to the in-vehicle system 11 regarding the user ID contained in another identification document of the user. When the in-vehicle system 11 receives the reading suggestion instruction regarding the user ID contained in another identification document, the sound output unit 14 of the in-vehicle system 11 outputs an audio message indicating that it is suggesting reading the user ID contained in the other identification document.

[0067] This allows users to be informed via voice when their user ID authentication using one form of identification fails, and when their user ID on another form of identification is to be read. Therefore, user convenience can be improved.

[0068] Furthermore, in the embodiment described above, if authentication of the user ID fails, the transmission unit 206 of the management device 20 sends a reading suggestion instruction to the mobile terminal device 30 regarding the user ID contained in another identification document of the user. When the mobile terminal device 30 receives the reading suggestion instruction regarding the user ID contained in another identification document, the suggestion unit 36 ​​of the mobile terminal device 30 displays an image indicating that it is suggesting the reading of the user ID contained in the other identification document.

[0069] This allows users to confirm on the screen of their mobile device 30 whether, if authentication of their user ID contained in one identification document fails, their user ID contained in another identification document will be read. Therefore, user convenience can be improved.

[0070] <Other Embodiments> In other embodiments, the count determination unit 204 of the management device 20 may determine whether at least one of the number of times facial recognition has failed and the number of times user ID authentication has failed is equal to or greater than a predetermined threshold. The predetermined threshold can be any value. If it is determined that at least one of the number of times facial recognition has failed and the number of times identification information authentication has failed is equal to or greater than a predetermined threshold, the transmission unit 206 of the management device 20 may notify a terminal device used by the administrator or the owner of the vehicle 10. The administrator may be the administrator of the authentication system 1 or the administrator of the vehicle 10.

[0071] By adopting this configuration, the administrator or the owner of the vehicle 10 can be aware of the fact that facial recognition or user ID authentication has failed. If a notification is sent to the terminal device used by the administrator, the administrator may authenticate the user of the vehicle 10 by additional authentication, such as telephone verification.

[0072] In other embodiments, the in-vehicle system 11 may include a camera for photographing the interior of the vehicle 10, a camera for photographing the exterior of the vehicle 10, and an anomaly detection unit. In this embodiment, the transmission unit 206 of the management device 20 transmits a monitoring instruction to the in-vehicle system 11 when it is determined that the number of times facial recognition has failed is above a threshold.

[0073] When the in-vehicle system 11 receives a monitoring instruction, the shooting control unit 111 of the in-vehicle system 11 instructs the shooting device that shoots the interior of the vehicle 10 and the shooting device that shoots the exterior of the vehicle 10 to shoot. The anomaly detection unit detects anomalies based on the images of the interior and exterior of the vehicle 10 generated by these shooting devices. The anomaly detection unit can detect anomalies by inputting the captured images into an anomaly detection model, which is a machine learning model. The anomaly detection model can be trained using captured images that represent various anomalies, such as images showing someone prying open a door or images showing someone breaking a door window. The sound output unit 14 emits an alarm sound when an anomaly is detected by the anomaly detection unit.

[0074] By adopting this configuration, if the number of failed facial recognition attempts exceeds a threshold, for example, if someone other than the owner or manager of the vehicle 10 attempts to get into the vehicle 10, an alarm can be sounded when an abnormality occurs inside or outside the vehicle 10.

[0075] Furthermore, in other embodiments, the in-vehicle system 11 may include a device for acquiring biometric information of a person sitting in the driver's seat of the automobile 10. For example, the in-vehicle system 11 may include a camera for photographing a person sitting in the driver's seat. In this embodiment, the camera control unit 111 of the in-vehicle system 11 instructs the camera to photograph the driver's seat. The transmission unit 114 of the in-vehicle system 11 transmits the photographed image of the driver's seat and the automobile ID of the automobile 10 in which the in-vehicle system 11 is installed to the management device 20.

[0076] When the management device 20 receives the captured image of the driver's seat and the vehicle ID, the facial recognition unit 201 of the management device 20 determines whether the biometric information of the user, based on the captured image of the user's face, matches the biometric information of the person sitting in the driver's seat. More specifically, the facial recognition unit 201 determines whether the user's face included in the image associated with the received vehicle ID matches the face of the person included in the captured image of the driver's seat. If it is determined that the user's face and the face of the person do not match, the transmission unit 206 of the management device 20 transmits an alarm instruction to the in-vehicle system 11. When the in-vehicle system 11 receives the alarm instruction, the sound output unit 14 of the in-vehicle system 11 emits an alarm sound.

[0077] Alternatively, the in-vehicle system 11 may include a device for acquiring fingerprint or voiceprint information of a person sitting in the driver's seat of the vehicle 10. The management device 20 may determine whether the fingerprint or voiceprint information of the user whose facial recognition was successful matches the fingerprint or voiceprint information of the person sitting in the driver's seat. More specifically, the management device 20 determines whether the fingerprint or voiceprint information of the user associated with the received vehicle ID matches the fingerprint or voiceprint information of the person sitting in the driver's seat.

[0078] By adopting this configuration, an alarm can be sounded if the person whose face recognition was successful is different from the person sitting in the driver's seat.

[0079] Furthermore, in another embodiment, the management device 20 may send information indicating a simple question to the user's mobile terminal device 30 before performing the facial recognition process. The user's mobile terminal device 30 sends information indicating the answer to the question entered by the user to the management device 20. The management device 20 may perform the facial recognition process if the answer entered by the user matches an answer that has been pre-associated with the question.

[0080] In the examples described above, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0081] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0082] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0083] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) An authentication system comprising: an in-vehicle system installed in a vehicle; a management device; and a portable terminal device, wherein the in-vehicle system comprises: a shooting device for shooting the face of a user outside the vehicle; and a transmitting unit for transmitting the image of the user's face captured by the shooting device to the management device; the management device comprises: a face recognition unit for performing face recognition processing using the captured image; a count determination unit for determining whether the number of times face recognition has failed is greater than or equal to a threshold based on the result of the face recognition processing; and a transmitting unit for sending a reading suggestion instruction for the user's identification information when it is determined that the number of times face recognition has failed is greater than or equal to the threshold; and the portable terminal device further comprises: a suggestion unit for suggesting the reading of the user's identification information when it receives the reading suggestion instruction. (Note 2) The authentication system as described in Note 1, wherein the in-vehicle system further includes a sound output unit that, upon receiving the read suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; the mobile terminal device comprises a read unit that, upon receiving the read suggestion instruction, reads the user's identification information contained in the user's identification document; and a transmit unit that transmits the read user's identification information to the management device; the management device further comprises an identification information authentication unit that performs authentication processing using the user's identification information; the transmit unit of the management device transmits a read suggestion instruction to the in-vehicle system regarding the user's identification information contained in another identification document of the user if authentication of the user's identification information fails; and the sound output unit, upon receiving a read suggestion instruction regarding the user's identification information contained in the other identification document, outputs a voice indicating that it suggests reading the user's identification information using the other identification document.(Note 3) The authentication system according to Note 1 or 2, wherein the mobile terminal device comprises a reading unit that reads the user's identification information contained in the user's identification document when it receives the reading suggestion instruction, and a transmitting unit that transmits the read user's identification information to the management device, the management device comprises an identification information authentication unit that performs authentication processing using the user's identification information, the transmitting unit of the management device transmits a reading suggestion instruction to the mobile terminal device for the user's identification information contained in another identification document of the user, and the suggestion unit, when it receives a reading suggestion instruction for the user's identification information contained in the other identification document, displays an image indicating that it suggests reading the user's identification information using the other identification document. (Note 4) The authentication system according to any one of Notes 1 to 3, wherein the count determination unit determines whether at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is equal to or greater than a predetermined threshold, and the transmission unit of the management device notifies the terminal device used by the administrator or the owner of the vehicle when it is determined that at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is equal to or greater than a predetermined threshold. (Note 5) The in-vehicle system further includes a sound output unit that, upon receiving the reading suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; the transmission unit of the management device transmits a monitoring instruction to the in-vehicle system when it is determined that the number of times the facial recognition has failed is above a threshold; the in-vehicle system includes a first imaging device for photographing the interior of the vehicle; a second imaging device for photographing the exterior of the vehicle; an imaging control unit that, upon receiving the monitoring instruction, instructs the first imaging device and the second imaging device to take a photograph; and an abnormality detection unit that detects abnormalities based on the images captured by the first imaging device and the second imaging device; and the sound output unit emits an alarm sound when an abnormality is detected by the abnormality detection unit, the authentication system according to any one of Notes 1 to 4.(Note 6) The authentication system according to any one of Notes 1 to 5, comprising: an audio output unit that, upon receiving the reading suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; the in-vehicle system comprising: a third imaging device that photographs the driver's seat of the automobile; and an imaging control unit that instructs the third imaging device to photograph the driver's seat; the transmitting unit of the in-vehicle system transmits the image of the driver's seat to the management device; the facial recognition unit of the management device determines whether the face of the user, for which facial recognition based on the image of the user's face has been successfully performed, matches the face of a person included in the image of the driver's seat; the transmitting unit of the management device, if it determines that the face of the user and the face of the person do not match, transmits an alarm instruction to the in-vehicle system; and the audio output unit of the in-vehicle system emits an alarm sound upon receiving the alarm instruction. (Note 7) A method comprising: an in-vehicle system taking a photograph of a user's face outside the vehicle; the in-vehicle system transmitting the photographed image of the user's face outside the vehicle to a management device; the management device performing a facial recognition process using the photographed image; the management device determining, based on the results of the facial recognition process, whether the number of failed facial recognition attempts is above a threshold; if the management device determines that the number of failed facial recognition attempts is above the threshold, sending a reading suggestion instruction for the user's identification information; and the user's mobile terminal device proposing to read the user's identification information upon receiving the reading suggestion instruction. (Note 8) The method according to Note 7, wherein when the mobile terminal device receives the read suggestion instruction, it reads the identification information of the user's identification document; the mobile terminal device transmits the read user's identification information to the management device; the management device performs authentication processing using the user's identification information; if authentication of the user's identification information fails, the management device transmits a read suggestion instruction to the in-vehicle system regarding the user's identification information contained in another identification document of the user; and when the in-vehicle system receives a read suggestion instruction regarding the user's identification information contained in the other identification document, it outputs a voice message indicating that it suggests reading the user's identification information using the other identification document.(Note 9) The method according to Note 7 or 8, wherein when the mobile terminal device receives the read suggestion instruction, it reads the identification information of the user's identification document, the mobile terminal device transmits the read user's identification information to the management device, the management device performs authentication processing using the user's identification information, and if the authentication of the user's identification information fails, the management device transmits a read suggestion instruction to the mobile terminal device for the user's identification information contained in another identification document of the user, and when the mobile terminal device receives a read suggestion instruction for the user's identification information contained in the other identification document, it displays an image indicating that it suggests reading the user's identification information using the other identification document. (Note 10) The method according to any one of the following items in Note 7 to 9, wherein the management device determines whether at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is above a predetermined threshold, and if the management device determines that at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is above a predetermined threshold, it notifies the terminal device used by the administrator or the owner of the vehicle. (Note 11) The method according to any one of Notes 7 to 10, wherein the management device determines that the number of times the facial recognition has failed is equal to or greater than a threshold, transmits a monitoring instruction to the in-vehicle system, the in-vehicle system, upon receiving the monitoring instruction, instructs a first imaging device for imaging the interior of the vehicle and a second imaging device for imaging the exterior of the vehicle to take photographs, the in-vehicle system detects an abnormality based on the images captured by the first imaging device and the second imaging device, and the in-vehicle system emits an alarm sound when an abnormality is detected.(Note 12) The method according to any one of Notes 7 to 11, wherein the in-vehicle system instructs a third camera that photographs the driver's seat of the automobile to photograph the driver's seat; the in-vehicle system transmits the image of the driver's seat to the management device; the management device determines whether the face of the user, for which facial recognition based on the image of the user's face has been successfully performed, matches the face of the person included in the image of the driver's seat; if the management device determines that the face of the user and the face of the person do not match, it transmits an alarm instruction to the in-vehicle system; and if the in-vehicle system receives the alarm instruction, it emits an alarm sound.

[0084] Some or all of the elements described in any appendix may apply to various hardware, software, recording means, systems, and methods for recording software.

[0085] 1: Authentication system 10: Automobile 11: In-vehicle system 12: ECU 13: Camera, shooting device 14: Speaker, sound output unit 15: Communication interface 16: Communication cable 20: Management device 21: Processor 22: Memory 23: Storage device 24: Communication interface 25: Bus 30: Mobile terminal device 31: Processor 32: Memory 33: Storage device 34: Communication interface 35: IC chip reader, reading unit 36: Display, suggestion unit 37: Bus 40: Network 110: Receiving unit 111: Shooting control unit 112: Sound output control unit 113: Unlock unit 114: Transmitter 120: Processor 121: Memory 122: Storage device 123: Communication interface 124 : Bus 200 : Receiving unit 201 : Face recognition unit 202 : Authentication result determination unit 203 : Count recording unit 204 : Count determination unit 205 : Identification information authentication unit 206 : Transmitting unit 300 : Receiving unit 301 : Display control unit 302 : Reading control unit 303 : Transmitting unit

Claims

1. An authentication system comprising: an in-vehicle system installed in a vehicle; a management device; and a portable terminal device, wherein the in-vehicle system includes: a camera for capturing images of a user outside the vehicle; and a transmission unit for transmitting images of the user's face captured by the camera to the management device; the management device includes: a face recognition unit for performing face recognition processing using the captured images; a count determination unit for determining whether the number of face recognition failures is greater than or equal to a threshold based on the results of the face recognition processing; and a transmission unit for transmitting a read suggestion instruction for the user's identification information when it is determined that the number of face recognition failures is greater than or equal to the threshold; and the portable terminal device further includes: a suggestion unit for suggesting the reading of the user's identification information when it receives the read suggestion instruction.

2. The authentication system according to claim 1, wherein the in-vehicle system further includes a sound output unit that, upon receiving the read suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; the mobile terminal device comprises a read unit that, upon receiving the read suggestion instruction, reads the user's identification information contained in the user's identification document; and a transmit unit that transmits the read user's identification information to the management device; the management device further comprises an identification information authentication unit that performs authentication processing using the user's identification information; the transmit unit of the management device transmits a read suggestion instruction to the in-vehicle system regarding the user's identification information contained in another identification document of the user if authentication of the user's identification information fails; and the sound output unit, upon receiving a read suggestion instruction regarding the user's identification information contained in the other identification document, outputs a voice indicating that it suggests reading the user's identification information using the other identification document.

3. The authentication system according to claim 1 or 2, wherein the mobile terminal device comprises a reading unit that, upon receiving the reading suggestion instruction, reads the user's identification information contained in the user's identification document, and a transmitting unit that transmits the read user's identification information to the management device, the management device comprises an identification information authentication unit that performs authentication processing using the user's identification information, the transmitting unit of the management device transmits a reading suggestion instruction to the mobile terminal device regarding the user's identification information contained in another identification document of the user, and the suggestion unit, upon receiving a reading suggestion instruction regarding the user's identification information contained in the other identification document, displays an image indicating that it suggests reading the user's identification information using the other identification document.

4. The authentication system according to any one of claims 1 to 3, wherein the count determination unit determines whether at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is equal to or greater than a predetermined threshold, and the transmission unit of the management device notifies the terminal device used by the administrator or the owner of the vehicle if it is determined that at least one of the number of times the facial recognition has failed and the number of times the authentication of the identification information has failed is equal to or greater than a predetermined threshold.

5. The authentication system according to any one of claims 1 to 4, wherein the in-vehicle system further includes a sound output unit that, upon receiving the reading suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; the transmission unit of the management device transmits a monitoring instruction to the in-vehicle system when it is determined that the number of times the facial recognition has failed is above a threshold; the in-vehicle system includes a first imaging device for photographing the interior of the vehicle; a second imaging device for photographing the exterior of the vehicle; an imaging control unit that, upon receiving the monitoring instruction, instructs the first imaging device and the second imaging device to take a photograph; and an abnormality detection unit that detects an abnormality based on the images captured by the first imaging device and the second imaging device; and the sound output unit emits an alarm sound when an abnormality is detected by the abnormality detection unit.

6. The authentication system according to any one of claims 1 to 5, wherein the in-vehicle system comprises: a sound output unit that, upon receiving the reading suggestion instruction, outputs a voice indicating that it suggests reading the user's identification information; a third imaging device that photographs the driver's seat of the automobile; and an imaging control unit that instructs the third imaging device to photograph the driver's seat; the transmission unit of the in-vehicle system transmits the image of the driver's seat to the management device; the face recognition unit of the management device determines whether the face of the user, for which face recognition based on the image of the user's face has been successfully performed, matches the face of a person included in the image of the driver's seat; the transmission unit of the management device, if it determines that the user's face and the face of the person do not match, transmits an alarm instruction to the in-vehicle system; and the sound output unit of the in-vehicle system emits an alarm sound upon receiving the alarm instruction.

7. A method comprising: an in-vehicle system capturing a photograph of a user's face outside the vehicle; the in-vehicle system transmitting the captured image of the user's face outside the vehicle to a management device; the management device performing a facial recognition process using the captured image; the management device determining, based on the results of the facial recognition process, whether the number of failed facial recognition attempts is above a threshold; if the management device determines that the number of failed facial recognition attempts is above the threshold, sending a read proposal instruction for the user's identification information; and the user's mobile terminal device, upon receiving the read proposal instruction, proposing to read the user's identification information.

8. The method according to claim 7, wherein when the mobile terminal device receives the read suggestion instruction, it reads the identification information of the user's identification document; the mobile terminal device transmits the read user's identification information to the management device; the management device performs authentication processing using the user's identification information; if authentication of the user's identification information fails, the management device transmits a read suggestion instruction to the in-vehicle system regarding the user's identification information contained in another identification document of the user; and when the in-vehicle system receives a read suggestion instruction regarding the user's identification information contained in the other identification document, it outputs a voice message indicating that it suggests reading the user's identification information using the other identification document.

9. The method according to claim 7 or 8, wherein, upon receiving the read suggestion instruction, the mobile terminal device reads the identification information of the user's identification document; the mobile terminal device transmits the read user's identification information to the management device; the management device performs authentication processing using the user's identification information; if authentication of the user's identification information fails, the management device transmits a read suggestion instruction to the mobile terminal device regarding the user's identification information contained in another identification document of the user; and, upon receiving the read suggestion instruction for the user's identification information contained in the other identification document, the mobile terminal device displays an image indicating that it suggests reading the user's identification information using the other identification document.

10. The method according to any one of claims 7 to 9, wherein the management device determines whether at least one of the number of times facial recognition has failed and the number of times authentication of the identification information has failed is equal to or greater than a predetermined threshold, and if the management device determines that at least one of the number of times facial recognition has failed and the number of times authentication of the identification information has failed is equal to or greater than a predetermined threshold, it notifies a terminal device used by the administrator or the owner of the vehicle.

11. The method according to any one of claims 7 to 10, wherein the management device determines that the number of times the facial recognition has failed is equal to or greater than a threshold, transmits a monitoring instruction to the in-vehicle system, the in-vehicle system, upon receiving the monitoring instruction, instructs a first imaging device for imaging the interior of the vehicle and a second imaging device for imaging the exterior of the vehicle to take photographs, the in-vehicle system detects an abnormality based on the images captured by the first imaging device and the second imaging device, and the in-vehicle system emits an alarm sound when an abnormality is detected.

12. The method according to any one of claims 7 to 11, wherein the in-vehicle system instructs a third camera that photographs the driver's seat of the automobile to photograph the driver's seat; the in-vehicle system transmits the image of the driver's seat to the management device; the management device determines whether the face of the user, for which facial recognition based on the image of the user's face has been successfully performed, matches the face of a person included in the image of the driver's seat; if the management device determines that the face of the user and the face of the person do not match, it transmits an alarm instruction to the in-vehicle system; and if the in-vehicle system receives the alarm instruction, it emits an alarm sound.

Images