Information processing method, information processing device, and program

Abstract

Provided are an information processing device, an information processing method, and a program with which it is possible to identify a customer from an acquired image with high accuracy and to improve convenience of the customer and quality of service. An information processing method according to a first aspect of the present invention is for processing information by using a coupon in which information about a service of a store is encoded, wherein a computer executes: a first step for reading a coupon code from a coupon when a customer uses the coupon in the store; a second step for acquiring an image that is obtained by photographing an object, which is the appearance of the customer or a license number of a vehicle that the customer drives, when the customer uses the coupon in the store; a third step for generating feature information of the object on the basis of the image that was acquired in the second step; and a fourth step for storing the coupon code that was read in the first step, the feature information that was generated in the third step, and a customer ID of the customer in a memory in association with each other.

Description

Information processing method, information processing apparatus and program

[0001] The present invention relates to an information processing method, an information processing apparatus, and a program that perform information processing using a coupon code obtained by encoding information related to store services.

[0002] A technique is known that identifies a person photographed by a camera by extracting facial features of the person included in an image captured by the camera and collating the extracted facial features with facial images of persons registered in advance in a database (see, for example, Patent Document 1). In recent years, with the development of image recognition technology using AI (Artificial Intelligence), it has become possible to identify people and objects with high accuracy based on images captured by cameras. Additionally, reading vehicle license plates in parking lots and drive-throughs and providing services to customers based on the read information is also being practiced.

[0003] Japanese Unexamined Patent Application Publication No. 2003-187229

[0004] The technology for automatically identifying a person from a face image captured by a camera can be applied to, for example, checking visitors at the entrance of a facility, and accepting registered users in counter services. Additionally, identifying and providing services to customers who repeatedly visit the same store (repeat customers) also leads to improved customer convenience and service quality.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide an information processing apparatus, an information processing method, and a program capable of highly accurately identifying a customer from an acquired image and improving customer convenience and service quality.

[0006] An information processing method according to a first aspect of the present invention is an information processing method that processes information using a coupon that codes information about a store's services, and is an information processing method in which a computer performs the following steps: a first step of reading a coupon code from a coupon when a customer uses a coupon at a store; a second step of acquiring an image of an object that is either the customer's appearance or the license plate number of the car the customer is riding in when the customer uses a coupon at a store; a third step of generating characteristic information of the object based on the image acquired in the second step; and a fourth step of associating the coupon code read in the first step, the characteristic information generated in the third step, and the customer's customer ID and storing them in memory.

[0007] An information processing device according to a second aspect of the present invention is an information processing device that processes information using a coupon that codes information about a store's services, and comprises: a reading unit that reads a coupon code from a coupon when a customer uses the coupon at a store; an acquisition unit that acquires an image of an object that is the appearance of the customer or the license plate number of the car the customer is riding in when a customer uses the coupon at a store; a feature generation unit that generates feature information of the object based on the image acquired by the acquisition unit; and a memory that stores the coupon code read by the acquisition unit, the feature information generated by the feature generation unit, and the customer's customer ID in association.

[0008] A third aspect of the present invention is a program that processes information using a coupon that codes information about a store's services, and causes a computer to execute the following steps: a first step of reading a coupon code from a coupon when a customer uses the coupon at a store; a second step of acquiring an image of an object that is either the customer's appearance or the license plate number of the car the customer is riding in when the customer uses the coupon at a store; a third step of generating characteristic information of the object based on the image acquired in the second step; and a fourth step of associating the coupon code read in the first step, the characteristic information generated in the third step, and the customer's customer ID and storing them in memory.

[0009] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that can perform highly accurate customer identification from acquired images and improve customer convenience and service quality.

[0010] Figure 1 is a diagram illustrating the configuration of a camera system to which the information processing device 1 according to this embodiment is applied. Figure 2 is a diagram showing an example of registration information 51 stored in the storage unit 150. Figure 3 is a diagram showing an example of camera information 52 stored in the storage unit 150. Figure 4 is a diagram showing an example of history information 53 stored in the storage unit 150. Figure 5 is a diagram showing an example of coupon issuance DB (database) 54 stored in the storage unit 150. Figure 6 is a diagram showing an example of terminal information 55 stored in the storage unit 150. Figure 7 is a diagram showing an example of the configuration of the terminal device 2. Figure 8 is a flowchart illustrating an example of the member registration process. Figure 9 is a flowchart illustrating an example of the coupon issuance process. Figure 10 is a flowchart illustrating an example of the process using a coupon. Figure 11 is a flowchart illustrating an example of the process when there is no coupon. Figure 12 is a flowchart illustrating an example of the process when new registration is performed. Figure 13 is a flowchart illustrating a first modified example of an embodiment of the present invention. Figure 14 is a flowchart illustrating the process by which the information processing device 1 changes the priority of images used for reference based on images and feature information received from the camera 220. Figure 15 is a flowchart illustrating the process of recording payment failure history in a second modified embodiment of the present invention (the embodiment of claim Y1). Figure 16 is a flowchart illustrating the process of executing irregular processing only for customers who meet specific requirements, based on past payment failure history stored in the storage unit 150 during payment.

[0011] Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same reference numerals will be used for identical components, and components that have already been described will be omitted from the description as appropriate.

[0012] (Information Processing Device) Figure 1 is a diagram illustrating the configuration of a camera system to which the information processing device 1 according to this embodiment is applied. The camera system shown in Figure 1 includes, for example, terminal devices 2 placed in each store, and an information processing device 1 that performs processing to identify objects such as people and things based on images captured by the camera 220 of the terminal device 2. The camera 220 is installed in a position where it can photograph specific objects (for example, people visiting a specific place), and performs processing to extract features of the specific object contained in the captured image (for example, features of a person's face or a car's license plate). The information processing device 1 identifies registered objects that correspond to the objects photographed by the camera 220 by comparing information about the features of objects generated by the camera 220 (feature information) with pre-registered feature information of objects. In the example of Figure 1, the multiple cameras 220 are connected to a network 9 such as the Internet or a LAN (Local Area Network), and the information processing device 1 acquires information from each camera 220 via the network 9. Note that the cameras 220 may be connected to the information processing device 1 without going through the network 9.

[0013] The information processing device 1 according to this embodiment processes information using coupons that encode information about store services. Coupons may include, for example, two-dimensional codes (QR codes®), barcodes, coupon numbers, etc. Coupons may be displayed on the screen of a mobile device, printed on paper, or represented as text data such as numbers and letters. Coupons may or may not include a customer ID, which is customer identification information. The customer ID is a unique value assigned when the coupon is issued.

[0014] In this embodiment, customers who use the same store multiple times are referred to as repeat customers, and their identification information is referred to as a repeat customer ID. The information processing device 1 performs information processing to provide services to customers using coupons, customer IDs, repeat customer IDs, etc.

[0015] In the example shown in Figure 1, the information processing device 1 includes a communication unit 110, an input unit 120, a display unit 130, an interface unit 140, a storage unit 150, and a processing unit 160.

[0016] The communication unit 110 is a device for communicating with other devices (such as the camera 220) via the network 9, and includes a device (such as a network interface card) that communicates in accordance with a predetermined communication standard such as Ethernet (registered trademark) or wireless LAN.

[0017] The input unit 120 is a device for inputting instructions and information in response to user operations, and includes any input device such as a keyboard, mouse, touchpad, or touch panel.

[0018] The display unit 130 is a device that displays an image corresponding to the image data input from the processing unit 160, and includes, for example, a liquid crystal display or an organic EL display.

[0019] The interface unit 140 is a device for inputting and outputting various data to the processing unit 160, and includes, for example, a general-purpose interface device such as USB, and a reader / writer device for recording media (e.g., memory card).

[0020] The storage unit 150 stores the program 50 executed by the processor of the processing unit 160, as well as data temporarily stored during the processing of the processing unit 160, data used in the processing of the processing unit 160, and data obtained as a result of the processing of the processing unit 160. For example, the storage unit 150 stores registration information 51 (Figure 2), camera information 52 (Figure 3), and history information 53 (Figure 4), which will be described later.

[0021] The storage unit 150 stores one or more programs 50 including instructions that can be executed by the processing unit 160, data temporarily stored during processing by the processing unit 160, data used in processing by the processing unit 160, data obtained as a result of processing by the processing unit 160, etc. The storage unit 150 may include, for example, a main memory (RAM, ROM, etc.) and an auxiliary memory (flash memory, SSD, hard disk, memory card, optical disc, etc.). The storage unit 150 may consist of one memory device or multiple memory devices. If the storage unit 150 consists of multiple memory devices, each memory device may be connected to the processing unit 15 via a computer bus or other arbitrary communication means.

[0022] The processing unit 160 comprehensively controls the overall operation of the information processing device 1 and executes predetermined information processing. The processing unit 160 includes, for example, one or more processors (CPU, MPU, etc.) that execute processing according to the instruction codes of one or more programs stored in the storage unit 150. The processing unit 160 operates as one or more computers by having one or more processors execute one or more programs in the storage unit 150.

[0023] The processing unit 160 may include one or more dedicated hardware components (such as an ASIC or FPGA) configured to implement specific functions. In this case, the processing unit 160 may perform the object identification process on one or more computers as described above, or it may perform at least a part of the process on the dedicated hardware components.

[0024] The processing unit 160 includes, for example, a code reading unit 161, a feature acquisition unit 162, a discrimination unit 163, a registration unit 164, a history update unit 165, and a camera management unit 166 as components that perform processing related to object identification, as shown in Figure 1.

[0025] The code reading unit 161 reads the coupon code from the coupon containing the customer ID. The coupon code is electronic information associated with the coupon. The code reading unit 161 reads the coupon code from the image information of the coupon captured by the camera 220 and extracts information about the store's services associated with the coupon code, as well as the customer ID contained in the coupon.

[0026] The feature acquisition unit 162 acquires feature information of an object that is generated based on an image taken by a camera 220 capable of photographing a specific object (such as a person or a car's license plate). The feature information is information about the features of the object (such as the facial features of a person) extracted from an image containing the object (an image taken by the camera 220). In this embodiment, as an example, the feature information is generated in the camera 220 that photographs the object. When the feature information is generated in the camera 220, the feature acquisition unit 162 may acquire the feature information directly from the camera 220, or it may acquire the feature information indirectly via one or more server devices (such as a cloud server) connected to the network 9. The feature information acquired by the feature acquisition unit 162 may be the same as the feature information generated by the camera 220, or it may be the same as the feature information generated by the camera 220, or it may be the same as the feature information that has been converted by a server device or the like interposed between the camera 220 and the information processing device 1.

[0027] The discrimination unit 163 compares the feature information acquired by the feature acquisition unit 162 (hereinafter sometimes referred to as "acquired feature information") with the feature information contained in the registration information 51 stored in the storage unit 150 (hereinafter sometimes referred to as "registered feature information"). The registered feature information contained in the registration information 51 is feature information that has been generated in advance for each object registered in the registration information 51, and is, for example, information obtained by extracting the features of an object from an image of the object that has been photographed in advance. Based on the comparison result between this registered feature information and the acquired feature information, the discrimination unit 163 identifies the object registered in the registration information 51 (hereinafter sometimes referred to as "registered object") that corresponds to the object contained in the image from which the acquired feature information was extracted (the image taken by the camera 220) (hereinafter sometimes referred to as "object in the image").

[0028] Figure 2 shows an example of registration information 51 stored in the memory unit 150. The registration information 51 includes predetermined information about each registered object. In the example in Figure 2, the object registered in the registration information 51 is a person. The reference numeral 510 in Figure 2 represents a set of information registered for one person. The set of information 510 shown in the example in Figure 2 includes a member ID assigned when registering as a member, a customer ID for identifying individual persons, an email address, a coupon code for a coupon assigned to the person, and a repeater ID used when the same store is used multiple times. The registration information 51 may also include the date of registration (registration date), personal information about the person (name, gender, date of birth, address, postal code of the address), a type assigned to the person (e.g., member type), current status (whether the registration is valid or not), data of the person's face image, and characteristic information showing the facial features of the person extracted from the face image. For each of the one or more registered persons, the registration information 510 includes a set of information 510 as shown in Figure 2.

[0029] The registration unit 164 processes the registration of information about the object to be registered (for example, a group of information 510 shown in Figure 2) in the registration information 51 of the storage unit 150. For example, when the code reading unit 161 reads a coupon code, the registration unit 164 processes the characteristic information acquired by the characteristic acquisition unit 162, links it with the customer ID, and stores it in the storage unit 150. For example, when a customer registers as a member, the registration unit 164 assigns and registers a member ID. Also, when a registered member visits a store for the first time and uses a coupon, the registration unit 164 assigns and registers a customer ID. Furthermore, when a customer visits a store, the registration unit 164 reads a coupon previously issued to the customer with the camera 220, and the code reading unit 161 extracts the customer ID contained in the coupon. When the customer's coupon is read, the characteristic acquisition unit 162 acquires the customer's characteristic information from the image of the customer's face taken by the camera 220. The registration unit 164 stores the customer characteristic information acquired by the feature acquisition unit 162 in the storage unit 150, linking it with the customer ID.

[0030] Furthermore, the registration unit 164 performs the process of adding information about the object to be registered to the registration information 51 in the storage unit 150. For example, the registration unit 164 displays a registration screen on the display unit 130 that prompts the user to input information about the object. The registration unit 164 adds the information entered by the input unit 120 to the registration information 51 according to the guidance on this registration screen. The registration unit 164 may also add information about the object provided by other devices that can communicate via the communication unit 110, or information about the object input from a recording medium or the like via the interface unit 140, to the registration information 51.

[0031] When the history update unit 165 determines that a registered object corresponding to an object in the image is identified by the identification unit 163, it stores information regarding the shooting history of the registered object in the storage unit 150. In this embodiment, as an example, camera information 52, which includes information about the camera 220, and history information 53, which is registered in the registration information 51, are stored in the storage unit 150.

[0032] Figure 3 shows an example of camera information 52 stored in the memory unit 150. In Figure 3, reference numeral 520 represents a set of information relating to one camera 220. The set of information 520 shown in the example in Figure 3 includes a camera ID for identifying each camera 220, a name assigned to each camera 220, the model of the camera 220, the functions that the camera 220 has (e.g., gender determination function, age determination function, facial expression detection function, etc.), the address of the camera 220 on the network 9 (e.g., IP address), the current status (whether or not it is connected to the network 9, etc.), the address of the location where shooting takes place, and the postal code of that address (shooting location postal code). The camera information 52 includes a set of information 520 as shown in Figure 3 for each of the one or more cameras 220 that the camera system has. The set of information 520 also includes a store ID, repeater ID, and characteristic information of the target object, which are used when registering or detecting repeaters.

[0033] Figure 4 shows an example of history information 53 stored in the memory unit 150. In the history information 53 shown in Figure 4, the object whose shooting history is recorded is a person (the same as the registration information 51 shown in Figure 2). The reference numeral 530 in Figure 4 indicates a group of information recorded when a person is identified by the discrimination process. The group of information 530 shown in the example in Figure 4 includes a product ID indicating the product purchased by the customer, a store ID where the photo was taken, a coupon code of a coupon presented by the person (customer) who was photographed, a time code indicating the date and time the photo was taken, a camera ID of the camera 220 that took the photo, a customer ID of the customer who was photographed, a repeat customer ID of the customer, characteristic information of the object obtained based on the photographed image, and a characteristic ID. The characteristic ID includes IDs linked to characteristic information based on the image of each object, such as a face ID linked to characteristic information based on the image of the customer's face taken by the camera 220, and a license plate ID linked to characteristic information based on the image of the license plate of a car taken by the camera 220.

[0034] Figure 5 shows an example of a coupon issuance DB (database) 54 stored in the storage unit 150. The reference numeral 540 in Figure 5 represents a group of information used when issuing coupons. The group of information 540 in Figure 5 includes a store ID indicating the store issuing the coupon, a terminal ID indicating the terminal device 2 issuing the coupon, an issuance target indicating the conditions for the person to whom the coupon will be issued, and postal code / GPS data indicating the region in which the coupon will be issued.

[0035] Figure 6 shows an example of terminal information 55 stored in the memory unit 150. The reference numeral 550 in Figure 6 represents a group of information relating to the terminal device 2. The group of information 550 in Figure 6 includes the store ID of the store where the terminal device 2 is installed, and a terminal ID for identifying each individual terminal device 2. The group of information 550 also includes a product ID and a coupon code used to determine the priority of product and service information to be displayed on the input / output unit 250 (see Figure 7) of the terminal device 2 when a customer uses the terminal device 2.

[0036] Each time an object is identified through the identification process, the history update unit 165 generates a set of information 530, as shown in Figure 4, based on the information contained in the registration information 51 for the identified object (such as customer ID) and the information contained in the camera information 52 for the camera 220 that photographed the identified object (such as camera ID and postal code of the shooting location), and adds this to the history information 53.

[0037] The camera management unit 166 performs processing related to the management of the cameras 220 provided by the camera system. For example, when adding a new camera 220 to the camera system, the camera management unit 166 displays a camera addition screen on the display unit 130 prompting the user to input information about the new camera 220. Following the instructions on this camera addition screen, the camera management unit 166 adds the information entered by the input unit 120 to the camera information 52 shown in Figure 3.

[0038] Figure 7 shows an example of the configuration of terminal device 2. Terminal device 2 is a product purchase terminal (for example, a kiosk terminal) or a tablet computer placed in a store. Terminal device 2 has a camera 220, a communication unit 210, an interface unit (I / F unit) 230, a storage unit 240, an input / output unit 250, and a processing unit 260.

[0039] The communication unit 210 is a device for communicating with other devices (such as the information processing device 1) via the network 9, and includes a device (such as a network interface card) that communicates in accordance with a predetermined communication standard such as Ethernet (registered trademark) or wireless LAN.

[0040] The camera 220 is a device for capturing images and includes an image sensor such as a CMOS image sensor, an optical system for guiding light from a subject to the image sensor, and an image processing device for processing the image captured by the image sensor.

[0041] The interface unit 230 is a device for inputting and outputting various data to the processing unit 260, and includes, for example, a general-purpose interface device such as USB, and a reader / writer device for recording media (such as memory cards).

[0042] The storage unit 240 stores programs to be executed by the processor of the processing unit 260, and also stores data temporarily stored during the processing of the processing unit 260, data used for processing by the processing unit 260, data obtained as processing results of the processing unit 260, and the like. The storage unit 240 includes, for example, a main storage device (ROM, RAM, etc.) and an auxiliary storage device (flash memory, hard disk, optical disk, etc.). The storage unit 240 may be configured of one storage device, or may be configured of a plurality of storage devices of one or more types.

[0043] The input / output unit 250 is a part that displays information and accepts input from a user. The input / output unit 250 is, for example, a touch panel display. Note that the input / output unit 250 may have a separate display unit and input unit.

[0044] The processing unit 260 is a device that comprehensively controls the overall operation of the camera 220, and includes, for example, one or more processors (CPU, MPU, etc.) that execute processing in accordance with the instruction codes of one or more programs 50 stored in the storage unit 240. The processing unit 260 operates as one or more computers when one or more processors execute one or more programs stored in the storage unit 240. The processing unit 260 may include one or more pieces of dedicated hardware (ASIC, FPGA, etc.) configured to implement specific functions. The processing unit 260 may execute all processes on one or more computers, or may execute at least part of the processes on dedicated hardware.

[0045] The processing unit 260 captures an image (still image or video) with the camera 220 and stores the captured data 31 in the storage unit 240. The processing unit 260 also generates feature information 32 about the features of a specific object contained in the image from the image captured by the camera 220 and stores it in the storage unit 240. If the object is a person and the features of the person's face are to be generated as feature information 32, for example, the processing unit 260 generates the feature information 32 based on feature points such as the person's eyes, nose, mouth, face, and contour in the image captured by the camera 220. The feature information 32 may be, for example, a multidimensional vector. In this case, the discrimination unit 163 may calculate the degree to which the facial features indicated by these feature information match based on the correlation coefficient between the vector indicated by the acquired feature information and the vector indicated by the registered feature information, and determine whether the person in the image matches the registered person by comparing the calculated degree of match with a predetermined threshold.

[0046] The processing unit 260 communicates with the information processing device 1 via the communication unit 210 and appropriately transmits the feature information 32 stored in the storage unit 240 to the information processing device 1. The processing unit 260 may also transmit information regarding the date and time the image in which the feature information 32 was generated was taken, along with the feature information 32, to the information processing device 1.

[0047] (Operation of the Information Processing Device: Information Processing Method) Here, the operation of the information processing device 1 according to the first embodiment having the configuration described above will be explained. Figure 8 is a flowchart illustrating an example of the member registration process. When a customer registers as a member, the information processing device 1 executes the process shown in the flowchart of Figure 8. First, the processing unit 160 receives the customer's email address (step S101) and password (step S102). For example, the customer enters their email address and password using application software (app) on their own terminal (such as a smartphone) or a terminal at the store, and the processing unit 160 of the information processing device 1 receives this entered email address and password.

[0048] Next, the processing unit 169 generates a member ID linked to the reception email address and password, and registers the member ID in the registration information 51 of the storage unit 150.

[0049] After generating the member ID, the processing unit 160 determines whether or not to issue a coupon (step S104). The processing unit asks for the customer's preference, and if the customer does not wish to receive a coupon (No in step S104), the processing ends after only completing member registration.

[0050] If the customer wishes to receive a coupon (Yes in step S104), the processing unit 160 generates a customer ID linked to the member ID, and registers the customer ID as registration information 51 in the storage unit 150 (step S105). When registering customer information, the processing unit may be configured to accept a postal code (a postal code corresponding to the customer's residence, workplace, or address of a frequently used station) or the customer's current position information (such as GPS data).

[0051] The processing unit 160 refers to the coupon issue DB 54 to generate a coupon, and registers information related to the generated coupon as registration information 51 in the storage unit 150 (step S106). The generated coupon includes the customer ID. Then, the processing unit 160 sends the generated coupon to the customer's terminal (step S107).

[0052] FIG. 9 is a flowchart for explaining an example of processing for issuing a coupon. The information processing apparatus 1 executes the processing shown in the flowchart of FIG. 9 every time a registered customer requests coupon issuance via their own terminal (such as a smartphone). The processing unit 160 sends information requesting input of an email address and password to the customer's terminal, and receives the email address (step S201) and the password (step S202) from the customer's terminal.

[0053] The processing unit 160, having received the email address and password, extracts the member ID from the email address and password, generates a customer ID linked to the member ID, and registers the customer ID as registration information 51 in the storage unit 150 (step S203). When registering customer information, the processing unit may be configured to accept a postal code (a postal code corresponding to the customer's residence, workplace, or address of a frequently used station) or the customer's current position information (such as GPS data).

[0054] The processing unit 160 generates a coupon by referring to the coupon issuance DB 54 and registers information related to the generated coupon as registration information 51 in the storage unit 150 (step S204). The generated coupon includes the customer ID. When issuing a coupon, it is preferable to generate a coupon that matches the information of the customer who requested the coupon issuance in the issuance target of the coupon issuance DB 54. For example, coupons for stores that match the postal code / GPS data in the coupon issuance DB 54 can be generated from the customer's postal code or GPS data. This makes it possible to generate coupons that can be used at stores that the customer frequently visits or that are close to their current location. Then, the processing unit 160 sends the generated coupon to the customer's terminal (step S205).

[0055] Figure 10 is a flowchart illustrating an example of processing using a coupon. When a customer uses a coupon issued in advance, the information processing device 1 executes the process shown in the flowchart of Figure 10. First, when a customer visits the store and operates the terminal device 2, or approaches the terminal device 2, the camera 220 of the terminal device 2 captures the customer's face, and the feature acquisition unit 162 acquires the feature information sent from the terminal device 2 (step S301).

[0056] Next, when the customer presents the pre-issued coupon to the camera 220, the camera 220 and the code reading unit 161 read the coupon code (step S302). Based on the coupon code, the code reading unit 161 extracts the customer ID contained in the coupon (step S303).

[0057] Next, the processing unit 160 obtains the repeater ID associated with the customer ID extracted from the coupon by the code reading unit 161 from the registration information 51 (step S304). If a repeater ID associated with the customer ID is not registered (for example, if the customer is visiting for the first time and presenting the coupon), a new repeater ID associated with the customer ID is assigned and registered in the registration information 51.

[0058] Next, the registration unit 164 associates the coupon code sent from the terminal device 2 with the customer's characteristic information and customer ID and registers them in the registration information 51 (step S305). Here, when registering customer characteristic information, for example, if characteristic information is obtained from an image of the customer's face, the face ID linked to that characteristic information is registered. If an image of the car's license plate is read when the customer visits the store by car, the license plate ID linked to the characteristic information obtained from the image of the license plate is registered as the characteristic ID. In addition, if the customer is a repeat customer who has visited the same store multiple times, the customer's characteristic information obtained each time they visit is registered and linked to the repeater ID. As a result, the amount of characteristic information linked to the repeater ID increases, and this becomes training data for the AI, improving the accuracy of customer identification. For example, even for the same customer, various image-based characteristic information is accumulated, such as the direction and expression of the customer's face when photographed by the camera 220, the presence or absence of glasses or a mask, and belongings.

[0059] Furthermore, if images of the customer's companions (including family, friends, and pets) are captured by camera 220, characteristic information about the companions may be registered in association with the customer ID. Since the coupon contains the customer ID, when the coupon code is read, the characteristic information of the captured object can be accurately matched with the customer ID, dramatically improving recognition accuracy.

[0060] Furthermore, if a customer has a companion, the system may determine whether the companion's customer ID is registered based on the companion's characteristic information, and if not, generate a companion ID for the companion. When registering a companion, the system may determine whether the companion is a family member or a friend of the customer based on the companion's characteristic information using attribute determination functions (age prediction, gender prediction, etc.) or AI machine learning using past registration information.

[0061] Next, the processing unit 160 acquires history information 53 linked to the customer ID (step S306). This allows the processing unit 160 to acquire the history of services provided to the customer who presented the coupon code (for example, past purchases, the date and time of visit based on the time code of a past photograph, and past coupon codes used).

[0062] Next, the processing unit 160 provides a service based on the acquired history information 53 (step S307). That is, the processing unit 160 displays service information on the input / output unit 250 of the terminal device 2 based on the service history of the history information 53 stored in the storage unit 150 in association with the repeater ID. For example, the processing unit 160 displays information on the input / output unit 250 about frequently purchased items (such as price, discount information, the display location of the item, and information on other related items) from the customer's purchase history at the store. Since repeaters tend to purchase similar items at the same time, displaying information about those items on the input / output unit 250 of the terminal device 2 makes it possible to process orders and payments quickly.

[0063] After the service is provided, the history update unit 165 updates the history information 53 (step S308). The history update unit 165 updates the history information 53 of the services received by the customer (such as the product ID of the purchased product and the coupon code used) linked to the repeater ID.

[0064] Figure 11 is a flowchart illustrating an example of processing when there is no coupon. When the customer operates or approaches the terminal device 2 without using a coupon code, the information processing device 1 executes the process shown in the flowchart of Figure 11.

[0065] First, when a customer visits the store and operates terminal device 2, or approaches terminal device 2, the camera 220 of terminal device 2 captures the customer's face, and the feature acquisition unit 162 acquires the feature information sent from terminal device 2 (step S401). If the customer visits the store by car, the camera 220 captures an image of the car's license plate, and the feature acquisition unit 162 acquires the feature information obtained based on the license plate image.

[0066] When the feature acquisition unit 162 acquires feature information based on the image of the customer's face or the car's license plate captured by the camera 220 (step S401), the discrimination unit 163 compares the customer's feature information (registered feature information) included in the registration information 51 with the acquired feature information of the person being photographed (step S402). If, as a result of comparing the acquired feature information and the registered feature information, no registered feature information corresponding to the photographed customer is found (No in step S403), the process proceeds to the new registration process shown in Figure 11. The new registration process will be described later.

[0067] If registered characteristic information corresponding to the photographed customer is found (Yes in step S403), the processing unit 160 obtains the repeater ID associated with that customer ID from the registration information 51 (step S404). When extracting the repeater ID from the registration information 51, if there are multiple characteristic information associated with the repeater ID, the matching is performed with priority given based on the shooting conditions at the time the image was taken. For example, images taken more recently are given a higher priority for matching. This allows it to be determined that a customer is a repeat customer even if they have not used a coupon. In addition, the shooting conditions include information about the shooting environment that enables highly accurate extraction of characteristic information used for matching, such as the angle of the face, the quality of the photograph, and the location from which characteristic information is extracted.

[0068] Next, the registration unit 164 links the customer characteristic information sent from the terminal device 2 to the customer ID and registers it in the registration information 51 (step S405). For example, if a customer is a repeat customer who has visited the same store multiple times, the customer characteristic information acquired each time they visit is registered in conjunction with the repeat customer ID. As the amount of characteristic information linked to the repeat customer ID increases, it becomes training data for the AI, improving the accuracy of customer identification.

[0069] Next, the processing unit 160 acquires history information 53 linked to the customer ID (step S406). This allows the processing unit 160 to acquire the history of services provided to the customer who presented the coupon (for example, past purchases, the date and time of visit based on the time code of a past photograph, and coupons used in the past).

[0070] Next, the processing unit 160 provides a service based on the acquired history information 53 (step S407). That is, the processing unit 160 displays service information on the input / output unit 250 of the terminal device 2 based on the service history of the history information 53 stored in the storage unit 150 in association with the repeater ID. For example, the processing unit 160 displays information on the input / output unit 250 about frequently purchased items (such as price, discount information, the display location of the item, and information on other related items) from the customer's purchase history at the store. Since repeaters tend to purchase similar items at the same time, displaying information about those items on the input / output unit 250 of the terminal device 2 makes it possible to process orders and payments quickly.

[0071] After the service is provided, the history update unit 165 updates the history information 53 (step S408). The history update unit 165 updates the history information 53 of the services received by the customer (such as the product ID of the purchased product and the coupons used) linked to the repeater ID.

[0072] Figure 12 is a flowchart illustrating an example of the process when a new registration is performed. If the answer is No in step S403 of Figure 11 (no matching registration characteristic information is found), the processing unit 160 executes the process shown in the flowchart of Figure 12. First, it determines whether or not to perform a new registration (step S501). A screen is displayed on the input / output unit 250 of the terminal device 2 operated by the customer, allowing the customer to choose whether or not to perform a new registration. If the customer does not wish to perform a new registration (No in step S501), the process ends. On the other hand, if the customer wishes to perform a new registration (Yes in step S501), the processing unit 160 sends information to the terminal device 2 requesting the input of an email address and password, and accepts the email address entered by the customer (step S502) and the password (step S503).

[0073] The processing unit 160, upon receiving the email address and password, generates a member ID linked to that email address and password, and registers the member ID as registration information 51 in the storage unit 150 (step S504). When registering customer information, the system may also accept postal codes (postal codes corresponding to the customer's place of residence, workplace, and frequently used train station addresses) and the customer's current location information (such as GPS data).

[0074] Next, the processing unit 160 generates a coupon and a customer ID linked to the member ID, and registers the generated coupon and customer ID as registration information 51 linked to the member ID in the storage unit 150 (step S405). In coupon generation, a coupon is generated based on the information of the newly registered customer if it matches the target of issuance in the coupon issuance DB 54. For example, a coupon for a store that matches the postal code / GPS data in the coupon issuance DB 54 is generated from the customer's postal code and GPS data. This makes it possible to generate coupons that can be used at stores close to places the customer frequently visits or their current location. The processing unit 160 sends the generated coupon to the customer's terminal (step S506).

[0075] Next, the processing unit 160 generates a repeater ID linked to the generated customer ID (step S507), and the registration unit 164 links the customer characteristic information sent from the terminal device 2 to the generated customer ID and registers it in the registration information 51 (step S508).

[0076] Next, the processing unit 160 provides services to the customer (step S509). That is, the processing unit 160 displays service information on the input / output unit 250 of the terminal device 2 based on the registered customer information. For example, the processing unit 160 displays information on the input / output unit 250 about high-priced products that the customer is likely to like (such as price, discount information, the display location of the product, and information on other related products). In the case of new registration, since history information 53 has not been registered, it becomes easier to provide services that meet the customer's needs by displaying recommended products and discount information at the store on the input / output unit 250 of the terminal device 2 based on the customer information obtained at the time of new registration.

[0077] After the service is provided, the history update unit 165 updates and registers the history information 53 (step S510). The history update unit 165 updates the history information 53 of the services received by the customer (such as the product ID of the purchased product and the coupon code used) linked to the repeater ID.

[0078] In the information processing method according to this embodiment, when a coupon is used, a feature ID (such as a facial ID or license plate ID) linked to the customer's characteristic information is comprehensively linked to the member ID, customer ID, and repeater ID, and this feature ID functions as the de facto master ID. To improve the accuracy of this feature ID (such as a facial ID or license plate ID) functioning as the master ID, each time a coupon is used, it is compared with the repeater ID linked to past history, and further, the image data in the database is ranked according to predetermined conditions (for example, in order of most recent shooting date and best shooting conditions) to assign a matching priority. This reduces the computational load energy of the computer and improves accuracy and matching speed. In addition, even if the initial member registration changes or the same customer has multiple member IDs, it is possible to retrieve past repeater history registered as the same customer using the feature ID (such as a facial ID or license plate ID) as the master ID, making it possible to continue providing services that enhance customer preferences and satisfaction. Furthermore, even if someone illegally obtains another person's member ID or obtains one through impersonation, the system can deter fraudulent use and impersonation by referencing the stored characteristic information, using a characteristic ID extracted from the characteristic information obtained when the user visited the store as the master ID. In other words, even if multiple user accounts are used in a single app, the user's behavior can be tracked using the repeater ID as the master ID, and even if a user account is lost, the user's activity history can be recovered, and fraudulent use of coupons can be prevented.

[0079] The following describes modifications of this embodiment. <First Modification> The information processing device 1 according to this embodiment (first modification) performs quality determination of acquired images and optimization processing (replacement determination) of existing feature information in order to achieve both improved matching accuracy and reduced computational energy load on the computer.

[0080] Figure 13 is a flowchart illustrating a first modified embodiment of the present invention. When an object is captured by an imaging unit such as a camera 220 (step S501), the processing unit of the camera 220 generates an image evaluation index indicating whether the image is suitable for use as an image for matching (step S502). The method for generating the image evaluation index will be described later.

[0081] The processing unit of camera 220 determines whether the image evaluation index satisfies certain conditions (step S503), and if it determines that it does, it generates feature information for the image (step S504). If the conditions are not met, it prompts re-imaging or stops processing based on the image. This prevents the generation and accumulation of unnecessary feature information from low-quality images and reduces the overall computational load on the system. The processing unit of camera 220 transmits the generated feature information and its image evaluation index to the information processing device 1. The information processing device 1 stores the feature information received from camera 220 and its image evaluation index in association with each other in the storage unit 150.

[0082] The image evaluation index described above and its generation method are explained below. The processing unit of camera 220 generates an image evaluation index for the image acquired in step S501 based on at least one of the frontality and clarity of the face in the image. The frontality (angle of the face) is defined so that the more the face of the object is facing directly towards camera 220, the higher the accuracy of feature extraction of eyes, nose, mouth, etc., and thus it can be determined to be more appropriate. Image clarity (photographic accuracy) index: This index quantifies the focus, low noise, and appropriate illumination (brightness) of the image, and is defined so that images with high resolution and contrast are evaluated as appropriate. The processing unit of camera 220 weights and sums each of the above indexes to generate an image evaluation index that quantifies the quality of the image.

[0083] Figure 14 is a flowchart illustrating the process by which the information processing device 1 changes the priority of images used for reference based on images and feature information received from the camera 220. When the processing unit 160 of the information processing device 1 receives an image (face image) and its feature information from the camera 220 (step S601), if the image and its feature information are already stored in the storage unit 150, it compares the image index information of the newly acquired image with the image index information of the feature information image already stored in the storage unit 150 (step S602).

[0084] If the image index information of the newly acquired image is high (superior) in the above comparison (step S603), the processing unit 160 changes the priority of the second feature information of the newly acquired image to be used preferentially in matching, rather than the first feature information already stored in the storage unit 150 (step S604). At this time, instead of changing the priority, the first feature information may be erased from the storage unit 150 and replaced with the second feature information. This reduces the capacity used by the storage unit 150.

[0085] According to the first modification, by determining whether the image evaluation index meets certain conditions (S503) before generating feature information, it is possible to prevent unnecessary feature generation processing (S504) from being performed on low-quality images unsuitable for matching. This eliminates unnecessary operations of the processing unit 160 and effectively reduces the computational load on the entire system. Furthermore, optimization of memory area and reduction of usage capacity: When the quality of a newly acquired image is better than that of an existing image, the old information is erased and replaced with new feature information (S604), making it possible to efficiently manage the finite storage capacity of the memory unit 150 and reduce the usage capacity. In addition, by prioritizing the use of high-quality images with high frontality and clarity, or images with excellent timeliness and comprehensive feature point extraction for matching (S604), the accuracy of matching can be dramatically improved. Furthermore, by focusing matching on high-quality data with low noise, the speed until matching is completed can be improved.

[0086] Furthermore, according to the first modification, the improved accuracy and speed of matching allow for quick identification of repeat customers even without presenting coupons. This reduces waiting times at the register and enhances customer convenience and satisfaction through faster ordering and payment processing. Additionally, by accumulating and updating characteristic information as AI training data and performing highly accurate identity verification, the fraudulent acquisition and impersonation of other people's member IDs can be effectively suppressed, ensuring the reliability of the service. Moreover, by accurately identifying customers based on the latest and most accurate characteristic information, it becomes possible to continuously provide high-quality services that meet customer needs, such as optimal product recommendations and discount information based on past purchase history.

[0087] Furthermore, the processing unit 160 may change the priority order to use preferentially in matching by further using the image acquisition time or feature point extraction comprehensiveness index of the image acquired in step S601. That is, the following indices may be used: Feature point extraction comprehensiveness index: Prioritizes images in which each part of the "feature point such as eyes, nose, mouth, face, and contour" is extracted with greater confidence and without being obscured by a mask or sunglasses. Temporal effectiveness (latest) index: To accommodate changes in appearance over time, if other image quality conditions are the same, images with a more recent shooting date and time are considered more appropriate.

[0088] <Second Modification> In this embodiment, when a customer fails to make a payment, the customer ID and history are recorded, and the customer is identified from the camera image when they next visit the store, and the history in memory is checked. Then, based on the past payment failure history, predetermined suggestions (such as offering alternative payment methods) are made to support smooth accounting.

[0089] Figure 15 is a flowchart illustrating the process of recording payment failure history in a second modified embodiment of the present invention (the embodiment of claim Y1). When a customer makes a payment at a store, the processing unit 160 of the information processing device 1 determines whether the payment was completed successfully or failed for any reason, based on the payment status obtained via the network 9 and the communication unit 110 (step S701).

[0090] If it is determined that the payment has failed (step S701), the processing unit 160 refers to the registration information 51 stored in the storage unit 150 to identify the customer ID (or repeater ID) of the customer in question. Then, it associates the identified ID with the history information that the payment failed and records it in the storage unit 150 as history information 53 (step ST702). This makes it possible to not only determine whether a payment is successful or not, but also to accurately store past payment trouble information linked to the attribute information of each customer.

[0091] Figure 16 is a flowchart showing how, at the time of payment, irregular processing is performed only for customers who meet specific requirements, based on past payment failure history stored in the memory unit 150. First, an image of the customer's appearance, etc., is acquired by an imaging unit such as a camera 220 at the payment site, and the discrimination unit 163 compares the feature information generated from the image with the registered feature information contained in the registration information 51 to determine whether or not the corresponding registration information exists (step ST801). If the corresponding registration information exists (step ST801), the processing unit 160 obtains a repeater ID, which is a unique identifier associated with the customer, from the registration information (step S802).

[0092] Next, the processing unit 160 uses the acquired repeater ID as a key to thoroughly check the customer's payment history stored in the history information 53 of the storage unit 150 (step S803). At this time, the processing unit 160 determines whether the past failure history included in the payment history satisfies specific judgment criteria (for example, cumulative number of failures) set in advance by the system (step S804). If it is determined that the judgment criteria are met (step S804), the processing unit 160 performs information processing to make an "irregular proposal" to the customer that is different from the usual (step S805).

[0093] This irregular suggestion includes processing such as displaying other recommended payment methods on the input / output unit 250, etc., with the aim of preventing register delays due to payment errors and supporting quick accounting. This achieves both the provision of appropriate services according to the customer's past payment trends and the maintenance of smooth accounting processing in the store. Furthermore, according to the second modification, customers who have previously caused payment errors can be identified in real time by image matching (ST801, S802), and the system can automatically switch from the standard payment flow to an individual processing flow including the irregular suggestion (S805), thereby suppressing the consumption of unnecessary system resources caused by processing stoppages and retries due to payment errors. In addition, by managing the history information of payment failures in a strong link with biometric identifiers (customer ID and repeater ID), a sophisticated data integrity and automatic control algorithm can be realized that accurately controls the trigger for exception processing for the same person regardless of changes in payment method. This prevents register delays due to payment errors, maximizes the accounting processing throughput of the entire system, and reduces the computational load on the computer. In other words, it allows for individual support for customers unfamiliar with payment methods, and also helps prevent fraudulent payments.

[0094] (Program) The program 50 according to this embodiment causes the computer to execute each step of the information processing method described above. The program 50 may be recorded on a computer-readable recording medium (optical disc, memory card, USB memory, or other non-temporary tangible medium). When the information processing device 1 is configured by a computer, the processing unit 160 may read at least a portion of one or more programs 50 recorded on such a recording medium using a recording medium reader (optical disc drive, etc.) or interface device (USB interface, etc.) not shown, and write it to the storage unit 150. Alternatively, the processing unit 160 may download at least a portion of one or more programs 50 from another device connected to a communication network using the communication unit 110 and write it to the storage unit 150. One or more programs 50 may include instructions that cause the processing unit 160 to execute at least a portion of the processing according to this embodiment, which will be described later.

[0095] As described above, this embodiment makes it possible to provide an information processing device, an information processing method, and a program that can perform highly accurate customer identification from acquired images and improve customer convenience and service quality.

[0096] It should be noted that the present invention is not limited to the embodiments described above, but includes various variations. For example, the embodiments described above include an example in which the facial features of a person included in the image captured by the camera 220 are extracted as feature information, but the features of any body part not limited to the face may be extracted as feature information, or the features of the entire body may be extracted as feature information.

[0097] Furthermore, while the above-described embodiment provides an example of identifying a person through discrimination processing, the object of discrimination processing may be a living being other than a human, or an object (non-living thing). For example, any moving object such as the exterior of a vehicle or a vehicle's license plate may be used as the object of discrimination processing.

[0098] Furthermore, while the above-described embodiment provides an example of generating object feature information in the camera 220, the present invention is not limited to this example. In other embodiments of the present invention, object feature information may be generated in a device other than the camera 220 (for example, an information processing device 1 or any server device connected to the network 9) based on the image captured by the camera 220.

[0099] 1...Information processing device, 2...Terminal device, 9...Network, 15...Processing unit, 31...Shooting data, 32...Feature information, 50...Program, 51...Registration information, 52...Camera information, 53...History information, 54...Coupon issuance DB, 55...Terminal information, 110...Communication unit, 120...Input unit, 130...Display unit, 140...Interface unit, 150...Storage unit, 160...Processing unit, 161...Code reading unit, 162...Feature acquisition unit, 163...Discrimination unit, 164...Registration unit, 165...History update unit, 166...Camera management unit, 210...Communication unit, 220...Camera, 230...Interface unit, 240...Storage unit, 250...Input / output unit, 260...Processing unit, 510, 520, 530, 540, 550...Information

Claims

1. An information processing method for processing information using a coupon that codes information about a store's services, the method comprising: a first step of reading a coupon code from a coupon when a customer uses the coupon at the store; a second step of acquiring an image of an object that is the appearance of the customer or the license plate number of the car the customer is riding in when the customer uses the coupon at the store; a third step of generating characteristic information of the object by image processing the image acquired in the second step; and a fourth step of storing the coupon code read in the first step, the characteristic information generated in the third step, and the customer's customer ID in a readable memory in association with each other.

2. The information processing method according to claim 1, further comprising: if the coupon has the customer ID attached to it, the first step of reading and identifying the customer ID from the coupon; and each time the coupon is read, storing the characteristic information generated in the third step in the memory in a readable manner, associated with the identified customer ID.

3. The information processing method according to claim 1, further comprising a sixth step of identifying the customer ID by comparing the feature information generated in the third step with the feature information stored in the memory, in a situation where the coupon is not read.

4. The information processing method according to claim 3, further comprising a first determination step of determining whether or not an image obtained in the second step satisfies certain conditions to be used as an image for matching, wherein the fourth step is to store in the memory an image which was determined to satisfy the certain conditions in the first determination step.

5. The information processing method according to claim 4, further comprising a second determination step of determining whether a newly acquired image is more suitable for the matching than the image on which the previously stored feature information was based, if the second determination step determines that the first feature information stored in the memory is positive, or replacing the first feature information stored in the memory with the second feature information generated from the newly acquired image, or using the second feature information preferentially over the first feature information in the matching.

6. The information processing method according to claim 5, wherein the first determination step determines whether the acquired image satisfies the certain conditions based on the degree of frontality to the object and the clarity of the image.

7. The information processing method according to claim 6, wherein the first determination step generates an image evaluation index based on at least one of the frontality and clarity of the face in the image obtained in the second step, and performs the determination based on the image evaluation index, and the second determination step performs the determination based on the image evaluation index.

8. The information processing method according to claim 7, wherein the second determination step is performed by further using at least one of the image acquisition time or the feature point extraction comprehensiveness index.

9. The information processing method according to claim 1, comprising: a seventh step of issuing a customer ID to the customer after the customer registration procedure of the customer; and an eighth step of issuing the coupon assigned to the customer ID after the seventh step, and downloading the coupon to the customer's terminal device, wherein the first step reads the coupon and obtains the coupon code and the customer ID issued in the seventh step.

10. The information processing method according to claim 2, further comprising a ninth step of issuing a repeater ID corresponding to the customer ID obtained in the first step, and storing the repeater ID in the memory in association with the customer ID, wherein in the fifth step, the characteristic information is stored in the memory in association with the repeater ID.

11. The information processing method according to claim 10, wherein in the fifth step, the history of services received by the customer is stored in the memory in association with the repeater ID.

12. The information processing method according to claim 11, further comprising a tenth step of obtaining the repeater ID linked to the customer ID extracted from the coupon code when the coupon is read in the first step, and displaying service information on a screen accessible to the customer based on the service history stored in the memory linked to the repeater ID.

13. The information processing method according to claim 11, further comprising an eleventh step of performing a process to compare the feature information generated in the third step with the feature information stored in the memory, and extract the repeater ID stored in the memory based on the comparison result for the feature information, in a situation where the coupon is not read.

14. If there are multiple pieces of characteristic information linked to the repeater ID, the information processing method according to claim 13, wherein in the 11th step, the matching is performed with priority given to the characteristic information based on the shooting conditions when the image was taken.

15. If there are multiple pieces of characteristic information associated with the repeater ID, the information processing method according to claim 13, wherein in the 11th step, priority is given to the pieces of characteristic information based on the shooting conditions when the image was taken, and the matching is performed with a higher priority given to the pieces of characteristic information with a more recent shooting time among the priority items.

16. The information processing method according to claim 10, wherein in step 9, the characteristic information generated based on the appearance of the customer or the license plate number of the car the customer is riding in is associated with the repeater ID.

17. The information processing method according to claim 1, wherein in the fourth step, characteristic information relating to the customer's companion is stored in the memory in association with the customer ID.

18. The information processing method according to claim 17, wherein in the fourth step, a companion ID of the companion is generated, and the characteristic information relating to the companion is stored in the memory in association with the companion ID.

19. The information processing method according to claim 1, further comprising an eleventh step of generating the customer ID of the customer when the customer uses the coupon at the store.

20. The information processing method according to claim 1, comprising the steps of: storing in memory the customer ID of a customer and the history of the failed payment in association with the customer when the customer fails to make a payment; identifying the customer ID from an image of the customer taken at the time of payment, and determining whether or not the customer has failed to make a payment in the past based on the history associated with the customer ID; and, if it is determined that the payment has failed, performing a process to make a predetermined irregular offer to the customer.

21. An information processing device that processes information using a coupon that codes information about a store's services, comprising: a reading unit that reads a coupon code from a coupon when a customer uses the coupon at the store; an acquisition unit that acquires an image of an object that is the appearance of the customer or the license plate number of the car the customer is riding in when the customer uses the coupon at the store; a feature generation unit that generates feature information of the object based on the image acquired by the acquisition unit; and a memory that stores the coupon code read by the acquisition unit, the feature information generated by the feature generation unit, and the customer's customer ID in association with each other.

22. A program for processing information using a coupon that codes information about a store's services, the program causing a computer to execute: a first step of reading a coupon code from a coupon when a customer uses the coupon at the store; a second step of acquiring an image of an object that is the appearance of the customer or the license plate number of the car the customer is riding in when the customer uses the coupon at the store; a third step of generating characteristic information of the object based on the image acquired in the second step; and a fourth step of associating the coupon code read in the first step, the characteristic information generated in the third step, and the customer's customer ID and storing them in memory.

Images