Monitoring system and display method

Abstract

This monitoring system comprises: a clerk monitor for displaying a state pertaining to payment for a product at each of a plurality of self-checkouts; and a control unit for controlling displaying of the clerk monitor. The clerk monitor displays, at one or more self-checkouts among the plurality of self-checkouts, that a customer paying for a product is away from a self-checkout when the customer is away from the self-checkout for a prescribed time or more.

Description

Monitoring System and Display Method

[0001] The present disclosure relates to a monitoring system and a display method.

[0002] In recent years, due to the reduction of store staff hours in retail stores and the like, the popularity of self-checkout has been rapidly advancing. In self-checkout, since the customer himself / herself scans the barcode of the product, it is important to take measures against illegal acts such as shoplifting where the product is taken away without paying for it.

[0003] For example, in Patent Document 1, when it is determined that a product has been taken out of the shopping basket, image data captured by a recognition camera is acquired and analyzed to determine whether the product has been put into a storage bag. When it is determined that the product has been put into the storage bag, it is determined whether the barcode of the product has been read and scanned by a barcode scanner between the shopping basket and the conducting wire of the storage bag. When it is determined that the product has not been scanned between the shopping basket and the conducting wire of the storage bag, an error warning is notified.

[0004] Japanese Patent Application Laid-Open No. 2011-054038, Japanese Patent Application Laid-Open No. 2024-082190, Japanese Patent Application Laid-Open No. 2024-103157

[0005] However, there is room for consideration regarding measures against illegal acts such as shoplifting.

[0006] The non-limiting embodiments of the present disclosure contribute to providing a monitoring system and a display method capable of appropriately taking measures against illegal acts such as shoplifting.

[0007] A monitoring system according to an embodiment of the present disclosure includes a monitor for store staff that displays a state related to the settlement of products at each of a plurality of self-checkouts, and a control unit that controls the display of the monitor for store staff. The monitor for store staff displays that the customer has left the self-checkout when the customer who settles the product at one or more of the plurality of self-checkouts has been away from the self-checkout for a predetermined time or more.

[0008] In one embodiment of the present disclosure, a monitor for store staff displays the status of product payment at each of a plurality of self-checkout registers, and the monitor for store staff displays that a customer who is paying for the products at one or more of the plurality of self-checkout registers has been away from the self-checkout register for a predetermined period of time or longer.

[0009] These comprehensive or specific embodiments may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium, or as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0010] According to one embodiment of this disclosure, appropriate measures can be taken against shoplifting and other fraudulent activities.

[0011] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features.

[0012] Figures illustrating the configuration of a self-checkout system, a diagram illustrating an example of a system configuration including multiple self-checkout registers, a diagram illustrating an example of the register's processing state and its transitions, a diagram illustrating the first example of the display transitions on a display, a diagram illustrating the second example of the display transitions on a display, a diagram illustrating the third example of the display transitions on a display, a diagram illustrating an example of area setting, a diagram illustrating an example of area setting, a diagram illustrating an example of shoplifting detection, a diagram illustrating an example of the display on the customer monitor in the first example of the notification method, a diagram illustrating an example of the display on the employee monitor in the first example of the notification method, a diagram illustrating an example of the display on the employee monitor in the second example of the notification method, a diagram illustrating another example of the display on the employee monitor in the second example of the notification method, a diagram illustrating an example of the display on the employee monitor in the third example of the notification method, a diagram illustrating another example of the display on the employee monitor in the third example of the notification method, a diagram illustrating an example of the display on the employee monitor in the fourth example of the notification method. A diagram showing an example of the display on the employee monitor in the fifth example of the notification method. A diagram showing an example of the display on the employee monitor in the sixth example of the notification method. A flowchart showing an example of the processing flow executed in the information processing device. A flowchart showing an example of the processing flow executed in the store terminal. A diagram showing the first example of the display on the employee monitor when the first example of non-payment detection is applied. A diagram showing the second example of the display on the employee monitor when the first example of non-payment detection is applied. A diagram showing the first example of the display on the employee monitor when the second example of non-payment detection is applied. A flowchart showing the first example of the processing flow for non-payment detection. A flowchart showing the second example of the processing flow for non-payment detection. A block diagram showing an example of the configuration of the information processing device 10. A block diagram showing an example of the configuration of the store terminal 200.

[0013] The embodiments of this disclosure will be described in detail below, with reference to the drawings as appropriate. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding for those skilled in the art.

[0014] The attached drawings and the following description are provided to enable a person skilled in the art to fully understand this disclosure, and are not intended to limit the subject matter described in the claims.

[0015] <Embodiment> In a self-checkout system where customers scan product barcodes themselves, shoplifting detection is performed when a customer takes an item from the product area where items are placed before scanning, and then moves it to the product area where scanned items are placed without scanning it. This failure to scan constitutes shoplifting. Failure to scan refers to the act of moving an item to the product area where scanned items are placed without scanning its barcode. Failure to scan may be replaced with terms such as scan error or not scanned.

[0016] In this shoplifting detection system, once an unscanned item has finished moving to the product display area, it is determined that a scan error has occurred. The system then notifies the customer of the scan error by displaying a message or other indication on a customer-facing display.

[0017] (System Configuration) Figure 1 is a diagram showing an example of the configuration of a self-checkout system. As shown in Figure 1, the self-checkout system includes an information processing device 1, a scanner 2, a camera 3, and a display 4. Figure 1 also shows the information processing device 10, camera 20, microphone 30, and display 40 according to this embodiment. In addition to the self-checkout system, Figure 1 also shows product display stands A1a and A1b. Hereinafter, the self-checkout system may be simply referred to as a "cash register" or a "POS (Point of Sales) register." Note that the information processing device 10, camera 20, microphone 30, and display 40 are examples of the configuration included in the monitoring system according to this embodiment.

[0018] Product display stand A1a is a stand on which products are placed before being scanned by scanner 2. Products placed on product display stand A1a before scanning may include shopping baskets or carts from the store containing the products before scanning.

[0019] Product display stand A1b is a stand on which products are placed after being scanned by scanner 2. The scanned products placed on product display stand A1b may include the customer's shopping basket or other container into which the scanned products are placed.

[0020] The information processing device 1 is, for example, a computer such as a personal computer or a server. In Figure 1, the information processing device 1 is placed on the product display stand A1a, but it may also be placed in the store's office. The information processing device 1 performs, for example, accounting processing for products.

[0021] Scanner 2 is a fixed-type scanner that is fixed to the product display stand A1a. Although not shown in Figure 1, a handheld scanner may also be provided.

[0022] Scanner 2 is connected to information processing device 1. Scanner 2 scans the product code attached to the product. Scanner 2 transmits the scanned product code to information processing device 1. The product code is a barcode, such as a JAN (Japanese Article Number) code. Scanner 2 may also be called a barcode reader or reader. The area where Scanner 2 scans the product code attached to the product is sometimes called the "scan area". The customer scans the product code by bringing the product close to the scan area and passing it through. Note that scanning the product code attached to the product may sometimes simply be called scanning the product.

[0023] Camera 3 is connected to the information processing device 1. Camera 3 transmits image data of the captured image to the information processing device 1.

[0024] Camera 3 is installed to photograph customers. However, camera 3 is not required.

[0025] The display 4 is connected to the information processing device 1. The display 4 displays, for example, product information of an item scanned by a customer using the scanner 2. The product information may include, for example, the name, price, and quantity of the product.

[0026] Display 4 may have a touch panel on its surface. The touch panel accepts customer input. For example, a customer can input information about products that do not have product codes, such as vegetables or fruits, via the touch panel and register them as purchased items. For example, if a customer purchases two apples, which do not have product codes such as barcodes, they can input the product name "apple" and the quantity "2" via the touch panel and register them as purchased items. Note that the device that accepts customer input is not limited to a touch panel. The device that accepts customer input may be, for example, a key input device separate from Display 4. Alternatively, customer input may be accepted using a smartphone or mobile device owned by the customer.

[0027] In the following, the actions of scanning product codes attached to products, and the actions of manually entering information for products that do not have product codes, may be collectively referred to as "product information registration" or "product registration."

[0028] Furthermore, in the following, the series of processes at the register from when a customer begins registering product information until they pay for the registered products will be referred to as "accounting processing." The process of registering product information will be referred to as "registration processing," and the process of paying for the registered products will be referred to as "payment processing."

[0029] Display 4 displays, for example, information corresponding to the status of the cash register. For example, the status of the cash register may include "waiting for accounting processing," "waiting for registration processing," "registration processing," and "waiting for payment." The status of the cash register may also be simply described as "cash register status."

[0030] The "checkout processing waiting" state indicates the cash register is waiting to process an transaction before a customer operates the register to complete the transaction. The "registration processing waiting" state refers to the state where a customer is operating the register to complete the transaction, and the register is waiting for the customer to register the product information. The "registration processing" state refers to the state within a specific time period from the moment the product information is registered. The "payment waiting" state refers to the state where the customer has completed registering the information for the products they wish to purchase, and the register is waiting to process the payment.

[0031] In addition to the above-mentioned register states, a "paid state" may also be added. This "paid state" indicates that the customer has made a payment while in the "payment waiting state." The "paid state" occurs when the register state transitions from the "payment waiting state" to the "accounting processing waiting state." For example, if a screen asking the customer whether they need a receipt or invoice is displayed on display 4 after the customer has paid, the register state transitions from the "payment waiting state" to the "paid state." After transitioning to the "paid state," if the register detects that the customer has left, the register state may transition from the "paid state" to the next state, the "accounting processing waiting state." Alternatively, if a screen indicating that the customer has placed a smartphone, credit card, prepaid card, or other payment card over a payment terminal set up on the register is displayed on display 4, the register state may transition from the "payment waiting state" to the "paid state." Alternatively, if a screen indicating that cash has been inserted into a designated location in the register (for example, the cash slot) or a screen indicating the amount of change is displayed on display 4, the register's status may transition from "waiting for payment" to "paid."

[0032] The transitions between each processing state of the cash register, and examples of displays on display 4 corresponding to the processing state of the cash register, will be described later.

[0033] The information processing device 1 performs accounting processing for the self-checkout system. For example, the information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (e.g., the results of product information input) via the touch panel on the surface of the display 4, and performs calculations for the total price of the products registered by the customer and payment processing for the amount paid by the customer.

[0034] The information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (for example, the result of inputting product information) via the touch panel on the surface of the display 4, and performs display control to display product information of the products registered by the customer on the display 4. In addition, the information processing device 1 performs display control to display information corresponding to the processing status of the register on the display 4 based on the customer operation information via the touch panel on the surface of the display 4.

[0035] Camera 20 is positioned to photograph the area around where the customer scans the product with scanner 2, and to photograph at least a portion of the display area of ​​display 4. For example, camera 20 is positioned above display 4. The field of view of camera 20 is set to include the area between product stands A1a and A1b, that is, the area around where the customer scans the product with scanner 2, and also include at least a portion of the display area of ​​display 4.

[0036] However, the camera 20 does not need to photograph the area where the customer scans the product with the scanner 2. In this case, the camera 20 is positioned to photograph at least a portion of the display area of ​​the display 4.

[0037] Note that the camera 20 is not limited to one. Multiple cameras 20 may be installed. For example, a camera 20 may be installed at a position that photographs the area around where the customer scans the product with the scanner 2, and at a position that photographs at least a portion of the display area of ​​the display 4. When cameras 20 are installed at these two positions, the camera 20 installed at the position that photographs the area around where the customer scans the product with the scanner 2 may be called an object detection camera 20, and the camera 20 installed at the position that photographs at least a portion of the display area of ​​the display 4 may be called a feature point detection camera 20. The camera image captured by the object detection camera 20 may be called an object detection camera image, and the camera image captured by the feature point detection camera 20 may be called a feature point detection camera image.

[0038] In addition, multiple cameras 20 may capture at least a portion of the display area of ​​the display 4, or multiple cameras 20 may capture the area around where the customer scans the product with the scanner 2.

[0039] The information processing device 10 is, for example, a computer such as a personal computer or a server. The information processing device 10 may be located in the office of the store. The information processing device 10 is connected to the camera 20 by wire or wireless and acquires camera images taken by the camera 20. The camera images may be still images, multiple still images in a sequence over time, or moving images.

[0040] Furthermore, the information processing device 10 may acquire sound from a microphone 30 installed around the self-checkout system. The microphone 30 acquires sound emitted from the self-checkout system and outputs the acquired sound to the information processing device 10. The sound emitted from the self-checkout system includes, for example, sounds corresponding to the processing status of the register, or sounds corresponding to the transition of the processing status of the register.

[0041] The information processing device 10 performs a determination process, for example, to determine whether or not there is a possibility of fraudulent activity (shoplifting) based on the acquired camera images and / or audio. The camera images and / or audio are examples of sensing results of information output by the self-checkout system.

[0042] For example, the information processing device 10 performs object detection processing based on the acquired camera image. The information processing device 10 determines the presence of an object and whether or not the object is moving through the object detection processing. Note that the object detection processing does not have to be based on the camera image. The information processing device 10 also determines the status of the cash register based on the acquired camera image and / or audio. Then, the information processing device 10 determines whether or not shoplifting has occurred based on the results of the object detection processing and the status of the cash register.

[0043] The information processing device 10 is connected to the display 40 and controls the display on the display 40.

[0044] The display 40 performs display based on the determination result in the information processing apparatus 10 under the control of the information processing apparatus 10. For example, the display 40 notifies customers of information regarding fraudulent acts. The notification of information regarding fraudulent acts includes, for example, the notification of a message indicating that a scanning omission has occurred. The notification of information regarding fraudulent acts may be replaced with a warning regarding fraudulent acts. Also, the fraudulent act here may correspond to shoplifting. Note that since the display 40 is used to notify customers, hereinafter, the display 40 may sometimes be referred to as a customer monitor.

[0045] In FIG. 1, an example in which the information processing apparatus 10 is provided separately from the information processing apparatus 1 of the self-checkout system is shown, but the present disclosure is not limited to this. The processing of the information processing apparatus 10 according to the present embodiment may be executed in the information processing apparatus 1 of the self-checkout system. Also, the information processing apparatus 1 of the self-checkout system may be connected to an external server by wire or wirelessly, and in the external server, the processing of the information processing apparatus 10 according to the present embodiment and / or a part of the processing of the information processing apparatus 1 may be executed.

[0046] Also, in FIG. (1), an example in which the camera 20, the microphone 30, and the display 40 are externally installed in the self-checkout system later is shown, but at least one of the camera 20, the microphone 30, and the display 40 may be built in the self-checkout system.

[0047] There are a plurality of self-checkouts as shown in FIG. 1 in the store. The plurality of self-checkouts communicate with a terminal for store clerks provided in the store and exchange information with each other.

[0048] FIG. 2 is a diagram showing an example of a system configuration including a plurality of self-checkouts. In FIG. 2, an information processing apparatus 10 and a store terminal 200 provided in each of the cash registers #1 to #N (N is an integer of 2 or more) are shown. The information processing apparatus 10 and the store terminal 200 communicate with each other via, for example, Wi-Fi (registered trademark).

[0049] The store terminal 200 may take the form of a tablet, smartphone, or other device, or it may be a stationary information processing device having a display, keyboard, touch panel, etc. The store terminal 200 is used by store staff to monitor the usage status of each self-checkout machine and shoplifting. The display of the store terminal 200 is used to notify store staff of information, and is therefore sometimes referred to as the "staff monitor" below. The display on the "staff monitor" may be executed by the control unit of the store terminal 200 (omitted in Figure 2), by the information processing device 10, or by the control unit of the store terminal 200 based on information notified from the information processing device 10.

[0050] The information processing device 10 transmits information to the store terminal 200, such as the processing status of the register, the number of times scans have been performed, and the number of times scan omissions have been detected. The information processing device 10 also transmits information to the store terminal 200 indicating whether or not there are people around each register. Furthermore, the information processing device 10 transmits images for display, such as camera images captured by the camera 20, to the store terminal 200.

[0051] The store terminal 200 transmits instruction information to the information processing device 10, which is generated based on instructions from the store staff. The instruction information includes instructions on the content of the information to be transmitted by the information processing device 10 to the store terminal 200, instructions on the frequency of transmission of the information to be transmitted by the information processing device 10 to the store terminal 200, and instructions regarding the settings of the information processing device 10. The information processing device 10 generates information to be transmitted to the information processing device 10, sets the transmission frequency, and changes the settings of the information processing device 10 according to the instruction information.

[0052] (Object detection processing) The object detection processing performed in the information processing device 10 will be described below.

[0053] The information processing device 10 detects (recognizes) products included in the image from the camera 20, for example, using a known image recognition technology (image recognition processing program) that utilizes artificial intelligence. In other words, the information processing device 10 detects products taken out from the product stand A1a through image recognition processing. The image recognition processing of products may be performed by another device, such as a server. Note that "detection" refers to the process of determining the existence of an object such as a product and identifying the location of that object, while "recognition" may refer to the process of identifying what the item included in the image is. However, in this embodiment, it is not necessary to accurately identify the product, and it may be difficult to accurately identify the product itself because the shooting angle and range of the camera 20 are limited. Therefore, in this embodiment, "detection" and "recognition" will not be distinguished unless there is a particular need to distinguish between them.

[0054] (Cash register processing status) The cash register processing status determined by the information processing device 10 is described below. As described above, the display 4 displays, for example, the cash register processing status. The cash register processing status includes "accounting processing waiting status", "registration processing waiting status", "registration processing status", and "payment waiting status".

[0055] Figure 3 shows an example of the processing states and transitions between them of a cash register. Figure 3 shows five processing states of a cash register and an example of the transitions between each state.

[0056] The accounting processing standby state (S11 in Figure 3) is the state in which the cash register is waiting to process an account before the customer operates the register to make a payment. For example, if, while in the accounting processing standby state, the customer gives an instruction to start the payment process via the touch panel on the display 4, the processing state of the cash register transitions from the accounting processing standby state to the registration processing standby state (S12 in Figure 3). The accounting processing standby state is an example of the state before payment.

[0057] When a customer scans the product code of a certain item while the register is in the registration processing waiting state, the register's processing state transitions from the registration processing waiting state to the registration processing state (S14 in Figure 3). Then, after a certain period of time has elapsed in the registration processing state, the register's processing state transitions from the registration processing state to the registration processing waiting state. The registration processing waiting state is an example of a state during checkout.

[0058] If, while in the registration processing waiting state, a customer instructs the register via the touch panel on display 4 to manually enter a product code in order to register a product that does not have a product code, the register's processing state transitions from the registration processing waiting state to the manual entry waiting state (S13 in Figure 3). The manual entry waiting state is an example of a state during checkout.

[0059] The "manual input waiting" state refers to a situation where a customer is operating the register to complete a transaction, and the register is waiting for the customer to manually enter the product information.

[0060] When the register is in manual input waiting state, if a customer manually enters information for a product that does not have a product code, the register's processing state transitions from manual input waiting state to registration processing state. Then, after a certain period of time has elapsed in the registration processing state, the register's processing state transitions from registration processing state to registration processing waiting state. The registration processing state is an example of a state during checkout.

[0061] If, while in the registration processing waiting state, the customer instructs the cash register to pay for the goods via the touch panel on display 4, the register's processing state transitions from the registration processing waiting state to the payment waiting state (S15 in Figure 3). The payment waiting state is an example of a state during checkout.

[0062] When the customer completes payment for the goods while the system is in the payment waiting state, the register's processing status transitions from payment waiting to accounting waiting. At the point where the system transitions from payment waiting to accounting waiting, the transaction is considered complete.

[0063] Furthermore, the registration processing waiting state, registration processing state, and manual input waiting state may be collectively referred to as the registration in progress state. In other words, the registration in progress state is a state in which the register processing state is neither payment waiting state nor accounting processing waiting state. Also, the register processing states described above are just examples, and this disclosure is not limited to these classifications.

[0064] In this embodiment, the information processing device 10 determines the processing status of the cash register, as shown in Figure 3, based on the acquired camera image and / or audio. Exemplarily, in this embodiment, the information processing device 10 determines the processing status of the cash register corresponding to the display on the display 4 by detecting feature points in the camera image captured from the display area of ​​the display 4.

[0065] Alternatively, instead of determining the register's processing status based on the acquired camera images and / or audio, as shown in Figure 3, the information processing device 10 may acquire information regarding the register's processing status from the self-checkout system. When acquiring information regarding the register's processing status from the self-checkout system, the information processing device 10 does not need to determine the register's processing status based on the acquired camera images and / or audio.

[0066] The following describes examples of displays on the display 4 corresponding to the processing status of the cash register. Note that the following examples are just examples, and this disclosure is not limited to these examples. Furthermore, the following shows the correspondence between the display examples on the display 4 and the processing status of the cash register shown in Figure 3, but this correspondence is just an example, and this disclosure is not limited to this. For example, the correspondence between the display examples on the display 4 and the processing status of the cash register may differ depending on the self-checkout system. For example, even if the display examples on the display 4 are common, the processing status of the cash register corresponding to the display examples may differ depending on the self-checkout system. Note that the correspondence between the display examples on the display 4 and the processing status of the cash register may be stored in the information processing device 10 in advance. Also, if the display examples change due to an update of the self-checkout system, the correspondence stored in the information processing device 10 may be updated.

[0067] (First display example of the display corresponding to the processing status of the register) Figure 4 is a diagram showing a first example of the display transition of display 4. Figure 4 shows display example 1 and display example 2 which transitions from display example 1. The display example transition shown in Figure 4 corresponds to an example of the transition from the registration processing waiting state to the registration processing state in Figure 3.

[0068] Display Example 1 is a display example corresponding to the registration processing waiting state. The display area shown in Display Example 1 includes an area for displaying a list of registered product names and prices, an area for displaying the number of registered products and the total price, an area corresponding to the "Payment button", and an area corresponding to the "Product Selection button".

[0069] The area displaying the list of registered product names and prices shows a list of registered product names and their corresponding prices. In the example in Figure 4, "Beef," "Eggs," "Milk," and "Tea" are registered products, and the product names and prices of each registered product are displayed in association with each other.

[0070] The area displaying the number of registered items and the total price will show the number of registered items and the total price. In the example in Figure 4, it will show that 4 items are registered and the total price for these 4 items is 1200 yen.

[0071] When the area corresponding to the "Payment button" is pressed by the customer, the registration process is completed and the system proceeds to the payment processing screen.

[0072] When the area corresponding to the "product selection button" is pressed by the customer, the system will transition to a screen for registering products that do not have a product code.

[0073] If the customer scans the product code for "coffee" while Display Example 1 is displayed, the display will transition from Display Example 1 to Display Example 2. Furthermore, if the customer scans the product code for "coffee" while Display Example 1 is displayed, a scanning sound may be emitted from the register's speaker.

[0074] Display Example 2 is an example of a display showing the registration processing status. The display area shown in Display Example 2 includes an area that displays the product name and price of the most recently registered product. Display Example 2 is an example of a display that indicates that the operation of scanning the product code attached to the product (in the example of Figure 4, "coffee") has been performed, and it shows the product that was recognized by the operation of scanning the product code.

[0075] The area displaying the product name and price of the most recently registered item will show the product name and price of the most recently registered item. In the example in Figure 4, since "coffee" was scanned most recently, the product name "coffee" and its price "200 yen" will be displayed.

[0076] Furthermore, if a certain amount of time has elapsed since the display example 2 was shown, a display corresponding to the registration processing waiting state, similar to the display example 1, will be shown.

[0077] For example, the information processing device 10 detects feature points in the camera image and, based on these feature points, distinguishes and determines whether the display on the display 4 corresponds to a registration processing waiting state as in Display Example 1, or to a registration processing state as in Display Example 2. For example, feature points that distinguish whether there are four areas arranged within the display area as in Display Example 1, or whether there is one area within the display area as in Display Example 2, may be detected. Alternatively, feature points that identify whether or not a "payment button" exists within the display area may be detected. In the example in Figure 4, the information processing device 10 detects feature points that can distinguish the difference between Display Example 1 and Display Example 2.

[0078] When the information processing device 10 detects a transition from a registration processing waiting state, as shown in display example 1, to a registration processing state, as shown in display example 2, it determines that the scan has been performed successfully.

[0079] Furthermore, if the information processing device 10 detects a scanning sound, it may determine that the register's processing state has shifted from the registration processing standby state to the registration processing state, and that the scan has been performed successfully.

[0080] Alternatively, instead of displaying an example like the one in Display Example 2, if scanning a product that requires age verification is being performed, a display indicating that there are restrictions on purchasing the product will be shown, consisting of an area for age verification and display areas corresponding to the "Yes" and "No" age verification buttons. The information processing device 10 may also detect feature points in the camera image and, based on these feature points, determine whether the display on the display 4 corresponds to the age verification display or the registration processing status.

[0081] As mentioned above, detecting that the register is in the registration processing state, that is, that the scan has been performed successfully, can be done by combining detection of the display on the display 4 corresponding to the registration processing state, detection of the scan sound, etc.

[0082] Furthermore, if a customer manually enters information about products without barcodes into the self-checkout system instead of registering products by scanning barcodes, the register's processing status transitions to a manual input waiting state. In this case, detection of the register's processing status being in the manual input waiting state may correspond to the detection of at least one image feature point on display 4, specifically the "list of product names for products without product codes" and the "screen display for entering quantities." It may also be possible to select which of the "list of product names for products without product codes" or the "screen display for entering quantities" is to be detected.

[0083] (Second display example of the display corresponding to the processing status of the register) Figure 5 is a diagram showing a second example of the display transition of display 4. Figure 5 shows display example 1 and display example 3 which transitions from display example 1. The display example transitions shown in Figure 5 correspond to an example of the transition from the registration processing waiting state to the payment waiting state in Figure 3. Note that display example 1 in Figure 5 is the same as display example 1 in Figure 4, so the explanation is omitted.

[0084] When a customer has finished registering the items they wish to purchase and presses the area corresponding to the "payment button" to begin payment for the registered items, the display on display 4 transitions from display example 1 to display example 3.

[0085] Display Example 3 is an example of a display corresponding to the payment waiting state. The display area shown in Display Example 3 includes an area for displaying buttons for selecting a payment method, and an area for displaying the number of registered items and the total amount.

[0086] When the information processing device 10 detects a transition from a registration processing waiting state, as shown in Display Example 1, to a payment waiting state, as shown in Display Example 3, it determines that it is waiting for customer payment. This payment waiting state may be just one example of a completed accounting state.

[0087] (Third display example of the display corresponding to the processing status of the cash register) Figure 6 is a diagram showing a third example of the display transition of display 4. Figure 6 shows display example 4 and display example 5 which transitions from display example 4.

[0088] Display Example 4 is an example of a display showing the accounting processing waiting state (S11 in Figure 3). The display area shown in Display Example 4 includes an area that displays a list of available payment methods and an area corresponding to the "start button" that is pressed when accounting processing is initiated.

[0089] When a customer presses the area corresponding to the "start button" to begin the accounting process, the display on display 4 transitions from display example 4 to display example 5.

[0090] Display Example 5 is a display that allows customers to choose whether or not to purchase a shopping bag to take home the items they have purchased. Display Example 5 is an example of a display that shows the checkout processing waiting state (S11 in Figure 3).

[0091] When a customer presses either selection button #1 or selection button #2 to input information about whether or not they want to purchase a shopping bag, the display on display 4 transitions from display example 5 to a different display.

[0092] When the information processing device 10 detects a waiting state for accounting processing, as shown in Display Example 4 or Display Example 5, it determines that it is waiting for the customer to start accounting processing (start of scanning). The start of accounting processing here corresponds to the accounting start state.

[0093] The information processing device 10 determines whether or not shoplifting has occurred (whether or not there were any scan errors) based on the results of the object detection process and the processing status of the cash register.

[0094] Here, in determining whether or not shoplifting has occurred (whether or not an item has been missed during scanning), it is detected whether or not the item has been moved to the product display stand A1b. This detection is performed in the object detection process described above. In this object detection process, an area is set for the camera images taken of the cash register and its surroundings.

[0095] The following is an example of setting an area in a camera image.

[0096] Figures 7A and 7B show examples of area settings. Figure 7A shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the starting area and ending area set within the shooting range. Figure 7BB also shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the personal belongings area and dead zone set within the shooting range.

[0097] The starting area is the area where the items to be scanned and the shopping basket containing those items are expected to be placed. The starting area corresponds to the area where the movement of the items to be scanned begins. The starting area includes at least the area of ​​product stand A1a. The starting area is set within the range of the register, including product stand A1a, but may include a margin beyond the range of the register.

[0098] The movement of an item may be determined to have begun when the customer picks up the item located in the starting area. Alternatively, the movement of an item may be determined to have begun when the position or size of the item in the starting area changes in the camera image.

[0099] The end area is the area where the shopping bags, reusable bags, baskets, etc., that will hold scanned items are expected to be placed. The end area corresponds to the area where the movement of items ends. The end area includes at least the area of ​​item placement stand A1b. The end area is set within the range of the register, including item placement stand A1b, but may include a margin beyond the range of the register.

[0100] The movement of the product to the end area may be determined to be complete when the product held in the customer's hand moves to the end area and the position and size of the product in the camera image do not change. Alternatively, the movement of the product to the end area may be determined to be complete when the product held in the hand moves to the end area and it is detected that the hand has released the product.

[0101] The personal belongings area is the area where a cashier (e.g., a customer) holding personal belongings is expected to be visible, and / or where a customer's personal belongings are expected to be placed. The personal belongings area is set outside the cashier's range in the camera image. The personal belongings area corresponds to the area where personal belongings that could be mistakenly identified as merchandise (e.g., a customer's wallet, smartphone, etc.) first appear. Note that the personal belongings area does not have to be set.

[0102] The dead zone is an area where it is not expected that a cashier operating a register with personal belongings (e.g., a customer) will be captured in the image, and / or where it is not expected that a customer's personal belongings will be placed. The dead zone is set outside the range of the register in the camera image. Because the dead zone may include images of adjacent registers or customers or store staff moving around, the dead zone corresponds to an area that can become noise. Items, people, etc. that appear in the dead zone are excluded from image processing such as product detection. Note that the dead zone does not have to be set.

[0103] Each area may be configured manually based on the camera image obtained when a camera 20 is installed at the cash register, the field of view of the camera 20 is set, etc.

[0104] Although not shown in Figures 7A and 7B, the area scanned by scanner 2 may be set as the scan area.

[0105] As mentioned above, here is an example of how to determine shoplifting when an area is defined.

[0106] Figure 8 shows an example of how shoplifting is detected. Figure 8 shows an example where a customer moves items in the self-checkout system shown in Figure 1.

[0107] The information processing device 10 detects the product (hereinafter referred to as product #x) that has started moving from the starting area. For example, the information processing device 10 detects the product held by the customer in the starting area as the product that has started moving from the starting area. Hereafter, the product that has started moving from the starting area may be referred to as product #x.

[0108] In the camera image, a frame may be superimposed on the image in the area of ​​product #x, as shown in Figure 8.

[0109] Furthermore, if the information processing device 10 detects that product #x has not been scanned, and also detects that the unscanned product #x has completed its movement to the end area, it determines that a scanning error has occurred for product #x. At this stage of determination, the movement of product #x to the end area is complete. For example, detection of the completion of product #x's movement to the end area corresponds to detection that product #x is stationary in the end area, or detection that the customer has released their hand from product #x in the end area.

[0110] <Example of notification method> To prevent shoplifting, it is desirable to display appropriate information on both the customer monitor (for example, the display 40 in Figure 1) and the employee monitor (for example, the display of the store terminal 200 in Figure 2). The conditions differ between the customer monitor and the employee monitor.

[0111] For example, it is desirable for customers to immediately notice if an item has been missed during scanning and to be able to scan it again. Therefore, it is desirable for customer monitors to notify customers clearly and quickly of items that have been missed during scanning. For example, it is desirable to notify customers of items that have been missed during scanning in real time.

[0112] For example, regarding employee monitors, it is desirable that they clearly notify employees of the status of each register so that employees can understand the status of each register. Furthermore, it is desirable that employee monitors also notify employees of past camera images and / or items that were missed during scanning.

[0113] Therefore, in this embodiment, when shoplifting is detected, the customer monitor displays an image of the item that was not scanned. When shoplifting is detected, the employee monitor normally displays the status of each item on the register and does not display an image of the item that was not scanned. However, when an employee performs an operation, the employee monitor displays an image of the item that was not scanned.

[0114] According to the notification method of this embodiment, appropriate displays can be shown on both the customer monitor and the employee monitor, enabling appropriate measures against shoplifting and other fraudulent activities.

[0115] <First Example of Notification Method> In the first example of the notification method, the customer monitor displays images of items that were not scanned when shoplifting is detected. For example, in the first example of the notification method, the customer monitor displays images of items that were not scanned when shoplifting is detected, regardless of whether the customer takes any action. The action taken by the customer may be an operation on the touch panel superimposed on the customer monitor, or an operation on various devices such as a keypad or buttons provided on the register being used by the customer. In other words, in the first example of the notification method, after displaying images of items that were not scanned, the customer monitor maintains the display of images of items that were not scanned, regardless of whether the customer takes any action, and the images of items that were not scanned are not hidden.

[0116] In the first example of the notification method, the employee monitor typically displays the status of each register and does not display images of items that were not scanned.

[0117] Figure 9 shows an example of the display on the customer monitor in the first example of the notification method. Figure 9 illustrates an example of the display when shoplifting is detected at register #1. Figure 9 shows an example of the display included in area A0, which is the display area of ​​the customer monitor at register #1.

[0118] As shown in Figure 9, area A0 includes area A1, which displays the image currently being captured by the camera; area A2, which displays a message regarding shoplifting detection; and area A3, which displays an image related to shoplifting detection. Area A2 displays a message prompting the customer to rescan if shoplifting is detected. Area A3 displays, for example, an image of a missed scan if shoplifting is detected, or an image detecting the most recent movement of an item if shoplifting is detected. Area A3 is displayed in area A0 depending on whether the customer has operated the customer monitor.

[0119] Here, the customer monitor may display a message such as "Welcome" when a new customer appears in front of the register. In this case, the message "Welcome" will be displayed when the system detects the presence of a customer in front of the register while in the "checkout processing waiting state". However, the message "Welcome" may not be displayed when the register transitions from the "checkout processing waiting state" to the "registration processing waiting state". By displaying a message such as "Welcome" when the presence of a customer in front of the register is detected, the customer is informed that the monitoring system according to this embodiment is aware of their actions and that store employees can confirm information regarding those actions, which helps to deter shoplifting.

[0120] Figure 10 shows an example of the display on the employee monitor in the first example of the notification method. Figure 10 shows an example of the information for each of the six registers, register #1 to register #6, being displayed on the employee monitor. Similar to Figure 9 described above, Figure 10 shows an example of the display when shoplifting is detected at register #1.

[0121] In the first example of the notification method, the employee monitor normally displays the status of each register and does not display images of items that were not scanned. Therefore, even if shoplifting is detected at register #1, as shown in Figure 10, information indicating that shoplifting was detected at register #1 is not displayed on the employee monitor, and the status of each register from register #1 to register #6 is displayed.

[0122] In Figure 10, the display area B0 of the employee monitor shows the status of each register #1 to #6. In Figure 10, the register status is shown, for example, indicating whether the transaction is before, during, or completed. For registers currently in the process of processing, the number of successful scans, the number of missed scans, and whether a customer is present near the register may also be displayed. In Figure 10, for example, registers #2, #5, and #6, which are in the process of processing, show the number of successful scans, the number of missed scans, and whether a customer is present near the register, respectively. The number of successful scans and missed scans are cleared, for example, when the start of the next transaction is detected. The information indicating whether a customer is present near the register may be displayed as "Paying" if a customer is present after all scans are complete. If a "Paid" status is added in addition to the register statuses described above, the employee monitor may display information indicating whether payment has been completed. For example, in the "Payment Pending" state, information indicating that payment is incomplete (e.g., the string "Payment Not Completed") may be displayed. After the information indicating that payment is incomplete is displayed, if the register status transitions from "Payment Pending" to "Payment Completed," the employee monitor may display information indicating that payment is complete (e.g., the string "Payment Completed").

[0123] According to the first example of the notification method, appropriate displays can be shown on customer monitors and employee monitors according to their respective needs, enabling appropriate measures against shoplifting and other fraudulent activities.

[0124] <Second example of notification method> In the second example of the notification method, the customer monitor displays images of items that were not scanned if shoplifting is detected, similar to the first example of the notification method described above.

[0125] In the second example of the notification method, the employee monitor displays images of unscanned items at a particular register when an employee performs a specific operation. Here, a specific operation by the employee includes, for example, an operation in which the employee designates a specific register. The employee's operation may be performed, for example, via a touch panel superimposed on the employee monitor, or via an operation unit other than the touch panel. When an employee performs an operation to designate a specific register, images of unscanned items at the designated register are displayed. If the employee does not perform a specific operation, the employee monitor displays the status of each register instead of displaying images of unscanned items.

[0126] Figure 11 shows an example of the display on the employee monitor in the second example of the notification method. Figure 11 shows an example of the information for each of the six registers, register #1 to register #6, being displayed on the employee monitor. Figure 11 also shows an example of the display when shoplifting is detected at register #5.

[0127] In the second example of the notification method, the employee monitor displays images of unscanned items at the relevant register when the employee performs a specific operation. For example, if shoplifting is detected at register #5, and the employee presses (or touches) area B1 which displays information for register #5, then, as shown in Figure 11, area B2 containing images of the unscanned items that were detected as shoplifting at register #5 will be displayed in area B0. Area B2 will display images of four items, n1 to n4, which are the unscanned items.

[0128] The number of items that were not scanned and displayed is not particularly limited. All items that have been determined to be not scanned may be displayed, or a limit may be placed on the number of items displayed, and only up to that limit may be shown. For example, the most recent number of items that were not scanned and up to the limit may be displayed. Alternatively, the images may be displayed one by one in chronological order.

[0129] Furthermore, if a store employee performs a specific operation and displays items that were not scanned for a specific register, information indicating that the specific operation was performed may be displayed. For example, this information may be displayed in the area that shows the status of the register for which the operation was performed, in the area that displays images of the items that were not scanned, or in both of these areas. The location in which this information is displayed may be chosen at the employee's discretion.

[0130] Figure 12 shows another example of the display on the employee monitor in the second example of the notification method. Figure 12 shows an example where information from each of the six registers, register #1 to register #6, is displayed on the employee monitor. Figure 12 also shows an example of the display when shoplifting is detected at register #5 and register #6.

[0131] In the example shown in Figure 12, when a store clerk performs a specific operation on each of the multiple cash registers, images of the items that were not scanned at each of those registers are displayed. For example, if shoplifting is detected at cash register #5 and cash register #6, and the store clerk presses (or touches) area B1 which displays the information for cash register #5 and presses area B3 which displays the information for cash register #6, then, as shown in Figure 12, area B2 containing images of the items that were detected as shoplifting and were not scanned will be displayed in area B0 at cash register #5, and area B4 containing images of the items that were detected as shoplifting and were not scanned will be displayed in area B0 at cash register #6. Area B2 displays images of four items, n1 to n4, which are items that were not scanned. Area B4 displays images of three items, n5 to n7, which are items that were not scanned.

[0132] The number of registers for which images of unscanned items are displayed on the same screen is not particularly limited. Images of all unscanned items from the register where the specific operation was performed may be displayed, or there may be an upper limit on the number of registers for which images are displayed, and images of unscanned items from up to that limit may be displayed. For example, the most recent up to the limit of unscanned items may be displayed. Alternatively, the images may be displayed one by one in chronological order.

[0133] According to the second example of the notification method, items suspected of shoplifting (for example, items that were not scanned) can be displayed in advance on the employee monitor, allowing employees to check the items and take appropriate measures against shoplifting and other fraudulent activities.

[0134] <Third Example of Notification Method> In the third example of the notification method, the employee monitor displays camera images taken a predetermined time prior to the time of shoplifting detection or the current time when shoplifting is detected. As a variation of the third example of the notification method, the employee monitor may have one of several modes selected, and the display may change according to the selected mode. For example, the display may change depending on the mode selected from a mode that displays images of items that were not scanned and a mode that displays camera images taken a predetermined time prior to the time of shoplifting detection or the current time. In this example, the predetermined time may be variable. For example, the predetermined time may be specified by the employee.

[0135] In the third example of the notification method, the customer monitor displays images of items that were not scanned when shoplifting is detected, similar to the first example of the notification method described above. In the third example of the notification method, when shoplifting is detected, the customer monitor displays images of items that were not scanned at the time the shoplifting was detected, and does not display images of items that were not scanned in the past.

[0136] Figure 13 shows an example of the display on the employee monitor in the third example of the notification method. Figure 13 shows an example of the information for each of the six registers, register #1 to register #6, being displayed on the employee monitor. Figure 13 also shows an example of the display when shoplifting is detected at register #5.

[0137] In the third example of the notification method, the employee monitor displays information according to the selected mode: a mode that displays images of unscanned items at the relevant register when the employee performs a specific operation, and a mode that displays camera images from a predetermined time prior to the shoplifting detection time or the current time at the relevant register. For example, if shoplifting has been detected at register #5, and the employee presses (or touches) area B1 which displays information for register #5, then, as shown in Figure 13, in the mode that displays images of unscanned items at the relevant register, area B2 containing images of the unscanned items that were detected as shoplifting at register #5 will be displayed in area B0. Area B2 will display images of four items, n1 to n4, which are unscanned items. Furthermore, for example, if shoplifting has been detected at register #5, and the store clerk presses (or touches) area B1 which displays information for register #5, then, as shown in Figure 13, in the mode that displays camera images from a predetermined time prior to the shoplifting detection time or the current time at the register in question, area B5 containing camera images from a predetermined time prior to the shoplifting detection time or the current time will be displayed in area B0 at register #5.

[0138] Figure 14 shows another example of the display on the employee monitor for the third example of the notification method. Figure 14 shows an example of the display when selecting the mode for register #5.

[0139] On the employee monitor, when an employee selects the mode for register #5, they press area B1, which displays information for register #5. This switches area B1 to area B6, which contains options for selecting the mode for register #5. If the employee selects option C1 displayed in area B6, area B2, which contains images of the unscanned items that were flagged as shoplifted, is displayed in area B0. If the employee selects option C2 displayed in area B6, area B7, which contains options for specifying the time of the camera image to display, is displayed. By selecting one of the options in area B7, area B5, which contains camera images from the time of the shoplifting detection or the current time, is displayed in area B0.

[0140] According to the third example of notification methods, not only can shoplifted items (for example, items that were not scanned) be identified, but camera images from the register and time period that the employee wants to check can also be viewed, allowing the employee to take more appropriate action.

[0141] <Fourth example of notification method> In the fourth example of notification method, if the register status does not change for a predetermined time between the start and end of the transaction, the employee monitor will display that the register status has not changed. Here, "no change in register status" corresponds to, for example, that no product scanning, manual entry of product information, or payment after product scanning has been performed.

[0142] In the fourth example of the notification method, the customer monitor displays an image of the unscanned item if shoplifting is detected, similar to the first example of the notification method described above. On the other hand, in the fourth example of the notification method, the customer monitor does not need to display that the register status has not changed.

[0143] Figure 15 shows an example of the display on the employee monitor in the fourth example of the notification method. Figure 15 shows an example where information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0144] In the example in Figure 15, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0145] In the example shown in Figure 15, during the checkout process, that is, from the start of checkout to the end of checkout, register #5 detects that there has been no change in the register's state for a predetermined period of time (for example, X seconds). Therefore, in the example shown in Figure 15, the message "No change in register state for X seconds after checkout started" is displayed for register #5, indicating that there has been no change in the register's state for a predetermined period of time (for example, X seconds).

[0146] According to the fourth example of notification methods, if a customer is unable to operate the register because they do not know how to do so (for example, how to scan items or how to pay), a store employee can quickly go to the customer and provide assistance, enabling appropriate measures to be taken against shoplifting and other fraudulent activities.

[0147] <Fifth example of notification method> In the fifth example of notification method, if it is detected that there is no one near the register for a specified period of time from the start to the end of the transaction, the employee monitor will display "The register in question is unoccupied."

[0148] In the fifth example of the notification method, the customer monitor displays an image of the unscanned item if shoplifting is detected, similar to the first example of the notification method described above. On the other hand, in the fifth example of the notification method, the customer monitor does not need to display that there is no one near the register for a specified period of time between the start and end of checkout. Alternatively, in the fifth example of the notification method, the customer monitor may display that there is no one near the register for a specified period of time between the start and end of checkout.

[0149] Figure 16 shows an example of the display on the employee monitor in the fifth example of the notification method. Figure 16 shows an example where information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0150] In the example in Figure 16, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0151] In the example shown in Figure 16, during the checkout process, that is, from the start of checkout to the end of checkout, register #5 detects that the customer is absent from the register for a predetermined period of time (for example, X seconds). Therefore, in the example shown in Figure 16, the message "Customer Absent" is displayed for register #5, indicating that the customer is absent from the register.

[0152] According to the fifth example of notification methods, even if shoplifting occurs, such as "scanning items but leaving the register without paying," store staff can be promptly notified that such an act may have occurred, allowing them to take prompt action against the act.

[0153] <Sixth example of notification method> In the sixth example of notification method, when shoplifting is detected, the store clerk can set whether to keep the customer monitor displaying the "shoplifting detection" status or to remove the "shoplifting detection" status. The setting to keep the customer monitor displaying the "shoplifting detection" status or to remove the "shoplifting detection" status may be set independently for each cash register, for example.

[0154] A shoplifting detection indicator is a display that informs a customer if there is suspicion of shoplifting. However, since it can be difficult to distinguish between actual shoplifting and actions that are not shoplifting but are mistakenly identified as such, such as scanning errors or false positives at the register, a "shoplifting detection indicator" may be a display that indirectly warns the customer. For example, a "shoplifting detection indicator" may notify the customer that "the most recently scanned item has not been scanned." For example, a message prompting the customer to "check if they have forgotten to scan any barcodes" may be displayed on the customer monitor.

[0155] Figure 17 shows an example of the display on the employee monitor for the sixth example of the notification method. Figure 17 shows an example where information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0156] The employee monitor displays images of items that were not scanned at the relevant register when an employee performs a specific operation. For example, if shoplifting is detected at register #5, and the employee presses (or touches) area B1 which displays information for register #5, area B2 containing images of the items that were not scanned and were flagged as shoplifting at register #5 will be displayed in area B0, as shown in Figure 17. Area B2 displays images of four items that were not scanned, from item n1 to item n4. Button B9 is also displayed in area B0 along with area B2. Button B9 is used to keep the customer monitor displaying the "shoplifting detection" status. When an employee presses button B9, the customer monitor display will remain in the "shoplifting detection" status. When button B9 is pressed, button B9 switches to button B10. Button B10 is used to remove the "shoplifting detection" status from the customer monitor display. Additionally, if a store employee presses button B10, the display on the customer monitor will be reset from the "shoplifting detection" state.

[0157] According to the sixth example of notification methods, if shoplifting is detected, appropriate measures can be taken to prevent shoplifting and other fraudulent activities.

[0158] <Method for hiding images of items that were not scanned> The above example of notification method showed an example where images of items that were not scanned are displayed on the customer monitor and / or the employee monitor. When images of items that were not scanned are displayed on the customer monitor and / or the employee monitor, the displayed images may be hidden based on the following conditions.

[0159] For example, on the customer monitor, if it is detected that an item that was not scanned has been scanned successfully and that the item has moved, the image of the item that was not scanned will be hidden. Furthermore, there does not need to be any operation on the customer monitor to hide the image of the item that was not scanned. In other words, the customer does not need to be permitted to hide the image of the item that was not scanned by operating the register's control panel (e.g., touch panel).

[0160] For example, if a store employee performs an operation via the control panel to hide the image of a product that was not scanned, which is displayed on the customer monitor of a specific register, the image of the product that was not scanned will be hidden on the customer monitor. The control panel used by the store employee may be, for example, a touch panel superimposed on the employee's monitor, or a device or user interface that receives the employee's input. The store employee can independently choose whether or not to hide the image of the product that was not scanned at each register. For example, by touching a designated area such as the register status display area or the image of the product that was not scanned via the touch panel, a separate screen will be displayed allowing the employee to choose whether or not to hide the image. By touching the corresponding area on the displayed selection screen, the store employee will hide the image of the product that was not scanned at the register corresponding to the touched area. For example, a separate screen such as "Do you want to hide the image?" may be displayed, and when the corresponding area is touched, the image of the product that was not scanned will be hidden.

[0161] (Example of the processing flow in the notification method of this embodiment) Next, a first example of the processing flow executed in the information processing device 10 will be described.

[0162] Figure 18 is a flowchart showing an example of the processing flow executed in the information processing device 10. The processing shown in Figure 18 is initiated, for example, when a customer who is about to pay for an item approaches the register and the display on the display 4 is detected to have changed in the camera image, or when an item is detected in the camera image. Whether or not a customer has approached the register may be determined, for example, based on whether a human body is visible in the camera image, or based on the detection result of a human presence sensor installed at the register. In the processing shown in Figure 18, the information processing device 10 acquires a camera image at predetermined intervals, performs object detection processing, and determines the processing status of the register.

[0163] The information processing device 10 determines whether or not all of the items to be purchased have been scanned (S101). For example, the information processing device 10 determines that all of the items to be purchased have been scanned if the status of the register indicates that the transaction is complete.

[0164] If all scans are complete (YES in S101), meaning there are no planned purchase items that have not yet been scanned, the flow ends.

[0165] If all scans are not complete (NO in S101), that is, if there are items to be purchased that have not been scanned, the information processing device 10 determines whether or not the scan for the item to be registered (hereinafter referred to as item #k) has been performed (S102). For example, the information processing device 10 determines that the scan for item #k has been performed if the register's processing status indicates the registration processing status.

[0166] If a scan is performed (YES in S102), the information processing device 10 determines whether or not it has detected that product #k has moved to the endpoint area (S103).

[0167] If it is not detected that product #k has moved to the endpoint area (NO in S103), the flow returns to S103, and object detection processing is performed until product #k moves to the endpoint area.

[0168] If the system detects that product #k has moved to the final area (YES in S103), the information processing device 10 controls the display of normal information on the customer monitor (S104). Then the flow returns to S101.

[0169] If a scan has not been performed (NO in S102), the information processing device 10 determines whether or not it has detected that product #k has moved to the endpoint area (S105).

[0170] If it is not detected that product #k has moved to the endpoint area (NO in S105), the flow returns to S101.

[0171] If the system detects that product #k has moved to the final area (YES in S105), the information processing device 10 executes a shoplifting detection process for product #k (S106).

[0172] If shoplifting is not detected for item #k (NO in S106), the flow returns to S101.

[0173] If shoplifting is detected for product #k (YES in S106), the information processing device 10 controls the display of a warning on the customer monitor indicating that shoplifting has been detected (S107).

[0174] Furthermore, the information processing device 10 controls the display of images of products that were not scanned on the customer monitor (S108). Then, the flow returns to S101.

[0175] Next, we will explain an example of the processing flow executed in the store terminal 200.

[0176] Figure 19 is a flowchart showing an example of the processing flow executed in the store terminal 200. The flow shown in Figure 19 starts, for example, when the store terminal 200 is powered on.

[0177] The store terminal 200 determines whether or not it has received display information to be displayed on the store terminal 200's display (store monitor) from the information processing device 10 (S201).

[0178] If no display information is received (NO in S201), the flow returns to S201.

[0179] If the display information is received (YES in S201), the store terminal 200 determines whether the register status included in the display information has been updated from the displayed state (S202).

[0180] If the register status has not been updated from the displayed state (NO in S202), the flow proceeds to S204.

[0181] If the register status has been updated from the displayed status (YES in S202), the store terminal 200 updates the displayed register status (S203). Then the flow proceeds to S204.

[0182] The store terminal 200 determines whether or not it has received an operation regarding the information to be displayed on the store terminal 200's display (store monitor) (S204). For example, the operations that can be accepted are one of the following: an operation to select a cash register, an operation to indicate that scan missed images should be displayed, an operation to indicate that past camera images should be displayed, etc.

[0183] If no operation regarding the information to be displayed is accepted (NO in S204), the flow returns to S201 without changing the information to be displayed.

[0184] If the terminal receives an operation regarding the information to be displayed (YES in S204), the store terminal 200 determines the target register according to the received operation and displays information about the target register on the display according to the received operation (S205). The flow then returns to S201.

[0185] <Notification Method Variation 1> The display may be changed depending on the number of detected scan omissions. Here, the number of detected scan omissions refers to the number of scan omissions detected while a single customer is scanning items for payment. For example, whether or not images of scanned items are displayed on the employee monitor may be determined based on the number of detected scan omissions. More specifically, if the number of detected scan omissions is high, the images of scanned items may be displayed more frequently than if the number of detected scan omissions is low. For example, if the number of detected scan omissions at a particular register exceeds a threshold, images of scanned items that were not scanned at that register may be displayed on the employee monitor. Note that regardless of whether the employee has performed an operation to display the scanned item images, if the number of detected scan omissions exceeds a threshold, images of scanned items that were not scanned at that register may be displayed on the employee monitor. In normal registration operations, the likelihood of frequent scan omissions is low, so if the number of detected scan omissions is high, there is a possibility that the customer is intentionally causing scan omissions for the purpose of shoplifting. Therefore, if there are many instances of detected scan errors, the system can be configured to display images of products that were not scanned more frequently, thereby alerting store staff.

[0186] <Variation 2 of Notification Method> For registers where shoplifting has not been detected, it is not necessary to display product images. For example, for registers where shoplifting has not been detected, even if a store employee performs an operation to display images of products that were not scanned at that register, the store employee's monitor does not need to display the product images (e.g., images of products that were not scanned). This reduces the burden on store employees to check the display, as it prevents the display of product images related to registers used by customers who are not likely to shoplift on the store employee's monitor, and allows for effective use of the limited display area of ​​the store employee's monitor. Here, "registers where shoplifting has not been detected" refers to, for example, registers where shoplifting has not been detected, but it may also include registers that do not have the shoplifting detection function itself, such as management self-checkout registers that can only be used by store employees.

[0187] <Variation 3 of Notification Methods> If a store employee performs an operation to display images of items that were not scanned at a register where shoplifting has not been detected, for example, if shoplifting has not been detected at register #X, which the employee has selected as the target for displaying the images of items that were not scanned, the employee monitor may display a message such as, "No shoplifting activity was detected in the current accounting process at register #X." In this case, the employee monitor may also display details of the register's status. For example, the elapsed time since the last scan or operation may be cited as details of the register's status. By displaying this information on the employee monitor, the employee can learn more about the circumstances under which a scan was missed, and thus confirm whether or not a misjudgment has occurred regarding shoplifting detection.

[0188] <4 Variations in Notification Methods> In the above example of notification methods (for example, the third example), whether or not to display images of products that were not scanned on the employee monitor can be selected independently for each register. For example, by selecting to display images of products that were not scanned for registers located in the blind spot of the employee, and not displaying images of products that are easily visible to the employee, the display area of ​​the employee monitor can be used efficiently. Also, in the above example of notification methods (for example, the third example), whether or not to display camera images from a predetermined time before the current time can be selected independently for each register. For example, for registers that the employee is paying close attention to, the situation can be accurately grasped without displaying past camera images, so by not displaying camera images from a predetermined time before, the display area of ​​the employee monitor can be used efficiently. Even if camera images from a predetermined time before are not displayed, the system may be controlled to display camera images from the current time.

[0189] <5 Variations of Notification Methods> In the above example of notification methods (for example, the fourth example), if the presence of a person around the cash register is detected, the action of the fourth example is performed. For example, if the presence of a person around the cash register is detected, and the status of the cash register does not change for a predetermined time between the start and end of the transaction, the employee monitor displays that the cash register status has not changed. If the presence of a person around the cash register is not detected, the employee monitor may not display that the cash register status has not changed. Since it is natural that the cash register status does not change when there is no one around the cash register, such as when a customer has left the cash register for some reason, performing this action can suppress excessive notifications to the employee. Existing person detection technology is used to detect whether or not a person is present. Existing AI (artificial intelligence) image recognition technology is used to distinguish whether the detected object is a person or not.

[0190] <Notification Method Variation 6> In the above examples of notification methods (for example, Examples 2 to 5), if multiple unscanned items are detected, the image of the unscanned item displayed on the employee monitor may be selected from among multiple unscanned item images. For example, this selection may be performed by the employee. For example, after one of the unscanned item images is displayed on the employee monitor, a message such as "Do you want to display images of other unscanned items?" is displayed, along with the options "Display" and "Do not display". If the employee selects "Display", the image of another unscanned item is displayed. If the employee selects "Do not display", the image of another unscanned item is not displayed. In this case, the employee monitor receives images of each of the multiple unscanned items. By allowing the employee to select which unscanned item image to display, the employee can switch which image of which item from among multiple unscanned items they prioritize checking.

[0191] <Variation 7 of Notification Methods> In the above examples of notification methods (for example, Examples 1 to 5), if it is detected that the scanned product image has been successfully scanned after the scanned product image is displayed on the customer monitor, the image of the product that was not scanned may be hidden. This allows the customer to return the register to a normal state without having to call a store employee and ask them to operate the register. Also, in the above examples of notification methods (for example, Examples 1 to 5), after the image of the product that was not scanned is displayed on the customer monitor, the display of the product that was not scanned may continue until a store employee performs a specific action. Here, the specific action may be, for example, an operation by the store employee to hide the display of the product that was not scanned, or an operation by the store employee to individually address the register where the display is shown. This means that the customer cannot hide the product that was not scanned without calling a store employee and asking them to operate the register, thus giving the store employee an opportunity to check the situation and more effectively deterring shoplifting.

[0192] <8 Variations in Notification Methods> In the above example of a notification method (for example, the fourth example), after the message "The register status will not change between the start and end of the transaction" is displayed on the employee monitor, the message will disappear if at least one of the following conditions is met. Condition 1: When the start of the next transaction is detected in the register status display area, the message "The register status will not change between the start and end of the transaction" will disappear. Condition 2: In a display other than the register status display area, if an employee performs an operation to remove the message "The register status will not change between the start and end of the transaction" as in the case of a product image that has not been scanned, the message will disappear.

[0193] In the above example of notification method (for example, example 5), after the message "The register is currently unoccupied" is displayed on the employee monitor, the message will be removed if at least one of the following conditions is met: Condition 1: When the start of the next transaction is detected in the register status display area, the message "The register is currently unoccupied" will be removed. Condition 2: In a display area other than the register status display area, if an employee performs an operation to remove the message "The register is currently unoccupied," similar to the case of a product image that was not scanned, the message will be removed.

[0194] <Notification Method Variation 9> In the above-mentioned examples of notification methods (for example, Examples 2 to 5), if the image of the product that was not scanned is displayed on the employee monitor, the time the scan failure occurred (for example, the time the scan failure was detected) may also be displayed. The time the scan failure occurred (for example, the time the scan failure was detected) may be replaced with the time the shoplifting occurred (for example, the time the shoplifting was determined). In addition, the image of the product that was not scanned and the time the scan failure occurred (for example, the time the scan failure was detected) may be displayed simultaneously.

[0195] <10 Variations of Notification Methods> Two or more of the above examples of notification methods may be used in combination or selectively. For example, both the fourth and fifth examples of notification methods may be used. For example, both the fourth and fifth examples may be used regardless of whether or not a store employee performs an operation. For example, if it is detected that no one is near the register for a predetermined time from the start to the end of the transaction, the employee monitor will display that "the register is unoccupied," and if the register status does not change for a predetermined time from the start to the end of the transaction, the employee monitor will display that the register status has not changed.

[0196] <Notification Method Variation 11> In the above-mentioned examples of notification methods (for example, Examples 1 to 5), if, after the completion of payment is detected, a person is detected near the register where the completion of payment was detected, the status of the register on the employee monitor may display "Payment in Progress".

[0197] <Notification Method Variation 12> In the above-mentioned examples of notification methods (for example, Examples 2 to 5), if images of products that were not scanned are displayed on the employee monitor, the images of products that were not scanned may be displayed one by one in chronological order. In this case, the images of products that were not scanned displayed in chronological order may include images of the same product that was not scanned taken at multiple times, or may include images of multiple products that were not scanned.

[0198] <13 Variations of Notification Methods> Two or more of the above examples of notification methods may be used in combination or selectively. For example, multiple examples may be applied simultaneously.

[0199] <14 Variations of Notification Methods> The above examples of notification methods (for example, Examples 1 to 5) may be applied not only when a scan of a product with a barcode is missed, but also when information on a product without a barcode is not entered.

[0200] <15 Variations of Notification Methods> In the above examples of notification methods (for example, Examples 1 to 5), the images of products that were not scanned and displayed on the employee monitor and / or customer monitor may be still images, videos, or both. The image of the product that was not scanned and displayed on one of the employee monitors or customer monitors may be a still image, while the image of the product that was not scanned and displayed on the other monitor may be a video.

[0201] <16 Variations in Notification Methods> The triggers for information to be sent from the register to the employee monitor may be changed. For example, the applicable trigger may be selected from the following triggers. This selection may be performed by the employee operating the control panel. Trigger 1: Notification is sent at regular intervals. Trigger 2: Notification is sent when shoplifting is detected, and no notification is sent if no shoplifting is detected.

[0202] <Notification Method Variation 17> Note that the above triggers may be set independently for each register. For example, when a store employee touches the display area on the touch panel that shows the status of each register, the options for trigger 1 and trigger 2 above may be displayed, and the setting may be changed by selecting as needed.

[0203] <Notification Method Variation 18> When the completion of a transaction is detected, the notification of information from the cash register to the employee monitor may be stopped. After the notification of information has been stopped, when the start of a transaction is detected, the notification of information from the cash register to the employee monitor may be resumed.

[0204] <Notification Method Variation 19> Before the start of payment is detected, the notification of information from the cash register to the employee monitor may be stopped.

[0205] <20 Variations in Notification Methods> In the above examples of notification methods (for example, Examples 1 to 5), when the start of payment is detected next, the shoplifting detection display may be deactivated regardless of whether the store clerk performs the prescribed operation on the clerk's monitor or not.

[0206] <Variation 21 of Notification Methods> In the above example of notification methods (for example, Example 6), if the number of times a scan omission is detected within the same transaction exceeds a predetermined number, the employee monitor may remain in a state where it is displayed that a scan omission has been detected. For example, if the number of times a scan omission is detected within the same transaction exceeds a predetermined number, the employee monitor may remain in a state where it is displayed that a scan omission has been detected, regardless of whether the employee has performed the predetermined operation via the control panel. If the number of times a scan omission is detected within the same transaction exceeds a predetermined number, there is a possibility that the customer is intentionally causing scan omissions, and there is a high possibility that shoplifting is occurring. Therefore, by maintaining the display that a scan omission has been detected, it is possible to more effectively prompt the employee to be vigilant.

[0207] <Variation 22 of Notification Method> When a specific person is making a purchase, the customer monitor may remain displaying that shoplifting has been detected, regardless of whether the store clerk performs a predetermined operation via the control panel. Here, the specific person may be a habitual shoplifter, etc. By setting habitual shoplifters, etc., as "specific persons," store clerks can focus their vigilance on "specific persons." For example, existing AI image recognition technology may be used to identify whether the person making the purchase is a "specific person."

[0208] <Variation 23 of Notification Methods> If there are no store employees around the employee monitor, the customer monitor may remain in a state where it is displayed that shoplifting has been detected, regardless of whether or not an employee performs a predetermined operation via the control panel. Existing AI image recognition technology may be used to identify whether or not an employee is around the employee monitor. For example, a warning can be issued if a person other than an employee is present around the employee monitor and attempts to operate a control panel that should be operated by an employee (e.g., a touch panel superimposed on the employee monitor). Here, a person other than an employee includes a person who is familiar with the store's operations and who intends to commit fraudulent acts (e.g., a habitual shoplifter). By maintaining the display on the customer monitor that shoplifting has been detected, if an employee who has temporarily left the location of the employee monitor returns to the location of the employee monitor, the employee can quickly and thoroughly grasp the situation. For example, if a customer commits shoplifting, but there are no store employees around the employee monitor, the shoplifting notification might disappear before an employee returns if the monitor is not kept showing that shoplifting has been detected. In such a case, it would be difficult for the employee to know that shoplifting has occurred. However, by keeping the shoplifting notification displayed, the employee can understand the situation.

[0209] <24 Variations of Notification Methods> In the embodiments described above, a notification method for shoplifting detection using camera images was shown. However, this disclosure may also apply to cases where shoplifting detection is performed using methods other than those using camera images. For example, this disclosure may also apply when shoplifting is detected using the following methods: - A shoplifting detection method using both camera images and a weight sensor installed on the product stand at the register. - A shoplifting detection method using a weight sensor installed on the product stand at the register. - A shoplifting detection method utilizing product information taken from shelves and customer movement history information. - A shoplifting detection method in which a security guard or other person in the store visually observes shoplifting-like behavior and notifies the customer from a terminal held by the security guard that shoplifting-like behavior has occurred.

[0210] In the shoplifting detection method using weight sensors installed on the product display stand at the register, weight sensors are installed on both the stand where products are placed before scanning and the stand where products are placed after scanning. The presence or absence of unscanned products is detected based on the difference in weight detected on the two stands. In the shoplifting detection method utilizing product information taken from shelves and customer movement history information, product information and customer movement history information are obtained by processing images from cameras such as surveillance cameras in the store. For example, shoplifting can be detected by checking which products each customer in the store picked up and put in their basket, and what route they took (for example, whether they left the store without going through the register). In these detection methods, images of unscanned products can be obtained by performing object detection on camera images collected immediately before the time when shoplifting (unscanned product) was detected.

[0211] <25 Variations of Notification Methods> In the above examples of notification methods (for example, Examples 1 to 6), information indicating whether or not the employee has viewed the image of the shoplifting detected may be displayed on the employee monitor. Whether or not the employee has viewed the image of the shoplifting detected is determined based on whether or not the employee has performed a specific operation to view the image of the shoplifting detected. For example, if the employee performs a specific operation, it is determined that the employee has viewed the image of the shoplifting detected. The specific operation is, for example, pressing the area on the employee monitor where the register status is displayed, as in the example above. By displaying information on the employee monitor indicating whether or not the employee has viewed the image of the shoplifting detected, the employee can be notified of any oversights, thus enabling more efficient shoplifting prevention. Note that the location on the employee monitor where this information is displayed is not particularly limited. For example, the display location may be at least one or both of the area that displays the status of each register and the area that displays the shoplifted item image. This allows the employee to recognize that there are unviewed images.

[0212] <26 Variations of Notification Methods> In the above-mentioned examples of notification methods (for example, Examples 1 to 6), information indicating the time when shoplifting was detected may be displayed on the employee monitor. The location on the employee monitor where this information is displayed is not particularly limited. For example, the display location may be at least one or both of the location that displays the status of each register and the location that displays images of the stolen items.

[0213] <Detection of Non-Payment> Shoplifting in stores includes not only taking items without scanning them and without paying for them, but also cases where items are scanned but the shoplifter takes them without paying for them.

[0214] This embodiment describes an example of detecting and appropriately notifying a customer when they scan an item but take it without paying for it. Note that the detection of an item being taken without payment is sometimes simply referred to as non-payment detection. For example, if a customer leaves the register while paying, there is a possibility of non-payment. On the other hand, even if a customer leaves the register while paying, if they do not take the item being scanned, the possibility of non-payment is low.

[0215] Therefore, in this embodiment, as an example of non-payment detection, it is detected whether or not a customer is present at the register between the start and end of the transaction. The presence or absence of a customer at the register is then notified to the employee's monitor, and further, it is detected whether or not there are any items remaining at the register, and the detection result is also notified to the employee's monitor.

[0216] Furthermore, the absence of a customer at the register corresponds to the absence of a customer in the area around the register where a customer is present during checkout. Similarly, the presence of goods at the register corresponds to the presence of goods in the area around the register where goods are present during checkout. The area where goods are present during checkout includes, for example, the area where goods are present before scanning, the area where goods are present after scanning, the area where goods are being scanned, and at least one of the areas where a customer can move with goods.

[0217] According to this embodiment, if a customer leaves the register while paying, appropriate information can be notified, allowing the store clerk to take appropriate action.

[0218] In the above-described embodiment, the customer monitor (for example, the display 40 in Figure 1) and the employee monitor (for example, the display of the store terminal 200 in Figure 2) may be included in the same system (for example, a monitoring system) or in different systems. For example, the employee monitor (for example, the display of the store terminal 200 in Figure 2) and the control unit that controls the employee monitor (for example, the control unit of the store terminal 200 in Figure 2 and / or the information processing device 10) may be included in one system (for example, an employee monitoring system), while the customer monitor (for example, the display 40 in Figure 1) may not be included in that same system (for example, an employee monitoring system). In other words, the customer monitor (for example, the display 40 in Figure 1) may or may not be included in the monitoring system according to this embodiment. Also, while the display control of the customer monitor as shown in the above-described embodiment is performed, the display control of the employee monitor may not be performed. Alternatively, the control of the customer monitor display and the control of the employee monitor display, as described in the above-described embodiment, may be performed in different systems (for example, different control units).

[0219] <Example of non-payment detection> In the first example of non-payment detection, if it is detected that no customer is near the register for a specified period of time from the start to the end of the transaction, a message indicating the customer's absence will be displayed on the employee monitor. Furthermore, if it is detected that no customer is near the register for a specified period of time, it will also be detected whether any items remain at the register, and the detection result will be notified to the employee monitor.

[0220] The items remaining at the register may be those in a basket, bag, or shopping bag, or they may not be in a basket, bag, or shopping bag. In other words, if the customer is absent, the system will detect whether or not a basket, bag, shopping bag, etc., remains at the register.

[0221] Figure 20 shows the first example of the display on the employee monitor when the first example of non-payment detection is applied. Figure 20 shows an example in which information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0222] In the example in Figure 20, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0223] In the example shown in Figure 20, during the checkout process, specifically from the start to the end of the checkout, register #5 detects that there are no customers near the register for a predetermined period of time. Register #5 also detects that the items remain in the register. Therefore, in the example shown in Figure 20, register #5 displays the message "Customer absent, items present," indicating that there are no customers near the register for a predetermined period of time and that the items remain in the register.

[0224] In the example shown in Figure 20, during the checkout process, that is, from the start of checkout until the end of checkout, register #6 detects that there are no customers near the register for a predetermined period of time. Register #6 also detects that there are no items remaining in the register. Therefore, in the example shown in Figure 20, register #6 displays the message "Customer absent, items present," indicating that there are no customers near the register for a predetermined period of time and that there are no items remaining in the register.

[0225] Figure 21 shows a second example of the display on the employee monitor when the first example of non-payment detection is applied. Similar to Figure 20, Figure 21 shows an example where information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0226] In the example in Figure 21, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0227] In the example shown in Figure 21, during the checkout process, specifically from the start to the end of the checkout, register #5 detects that there are no customers near the register for a predetermined period of time. Register #5 also detects that the items remain in the register. Therefore, in the example shown in Figure 21, register #5 displays the message "Customer absent, items present," indicating that there are no customers near the register for a predetermined period of time and that the items remain in the register, along with the message "Low probability of non-payment," indicating that the likelihood of non-payment is low.

[0228] In the example shown in Figure 21, during the checkout process, specifically from the start to the end of the checkout, register #6 detects that there are no customers near the register for a predetermined period of time. Register #6 also detects that there are no items remaining in the register. Therefore, in the example shown in Figure 21, register #6 displays the message "Customer absent, items present," indicating that there are no customers near the register for a predetermined period of time and no items remaining in the register, along with the message "High probability of non-payment," indicating that this situation likely indicates unpaid bills.

[0229] For example, if a store employee sees the display shown in Figure 20 or Figure 21, they may take immediate action to find the person who committed the shoplifting by rushing to register #6 or by informing other staff members that there is a high probability that shoplifting has occurred.

[0230] The messages shown for cash registers #5 and #6 in Figure 20 or Figure 21 may be displayed regardless of whether the store clerk operates the store clerk monitor or not.

[0231] In the first example of non-payment detection, information about customer behavior is notified to employee monitors, allowing employees to take appropriate action.

[0232] <Second example of non-payment detection> In the second example of non-payment detection, customer movement history information is maintained. Here, customer movement history information includes the direction of movement, distance traveled, and time of movement. The history of direction of movement, distance traveled, and time of movement are collectively referred to as movement history. For example, customer movement history information includes the movement history until the customer reaches the register, the movement history while operating the register, and the movement history when leaving the register.

[0233] In the second example of non-payment detection, if the absence of a customer is detected in the first example, information about the direction the customer moved is further notified to the employee monitor. For example, information such as whether the customer moved towards the entrance / exit, or towards the product shelves or other directions, is notified to the employee monitor. In this second example, the direction the customer moved also influences the determination of whether there is a high or low probability of non-payment, and the content displayed on the employee monitor changes according to the direction the customer moved.

[0234] Figure 22 shows the first example of the display on the employee monitor when the second example of non-payment detection is applied. Figure 22 shows an example in which information for each of the six registers, from register #1 to register #6, is displayed on the employee monitor.

[0235] In the example in Figure 22, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0236] In the example in Figure 22, during the checkout process, that is, from the start to the end of the checkout, register #5 detects that there are no customers near the register for a predetermined period of time. Furthermore, register #5 detects that the items remain in the register. Therefore, in the example in Figure 22, register #5 displays the message "Customer absent, items present," indicating that there are no customers near the register for a predetermined period of time and that the items remain in the register. Additionally, in the example in Figure 22, the movement history information of customer X, who was paying at register #5, indicates that customer X moved in a direction other than the entrance / exit. Therefore, register #5 displays a message indicating that customer X, who was paying at register #5, moved in a direction other than the entrance / exit.

[0237] In the example in Figure 22, during the checkout process, that is, from the start to the end of the checkout, register #6 detects that there are no customers near the register for a predetermined period of time. Furthermore, register #6 detects that there are no items remaining in the register. Therefore, in the example in Figure 22, register #6 displays the message "No customer, no items," indicating that there are no customers near the register for a predetermined period of time and that there are no items remaining in the register. Additionally, in the example in Figure 22, the movement history information of customer Y, who was paying at register #6, indicates that customer Y moved towards the exit. Therefore, register #6 displays a message indicating that customer Y, who was paying at register #6, moved towards the exit.

[0238] In the example in Figure 22, during the checkout process, that is, from the start to the end of the checkout, register #2 detects that there are no customers near the register for a predetermined period of time. Furthermore, register #2 detects that there are no items remaining in the register. Therefore, in the example in Figure 22, register #2 displays the message "No customer, no items," indicating that there are no customers near the register for a predetermined period of time and that there are no items remaining in the register. Additionally, in the example in Figure 22, the movement history information of customer Z, who was paying at register #2, indicates that customer Z moved in a direction other than the entrance / exit. Therefore, register #2 displays a message indicating that customer Z, who was paying at register #2, moved in the direction of the entrance / exit.

[0239] Figure 23 shows a second example of the display on the employee monitor when the second example of non-payment detection is applied. Similar to Figure 22, Figure 23 shows an example where information for each of the six registers, register #1 to register #6, is displayed on the employee monitor.

[0240] In the example shown in Figure 23, the status of cash registers #1 and #3 is "before payment," while the status of cash registers #2 and #4 through #6 is "in progress." The "in progress" state corresponds to the state from before payment begins until payment is completed.

[0241] In the example in Figure 23, during the checkout process, that is, from the start to the end of the checkout, register #5 detects that there is no customer near the register for a predetermined period of time. Furthermore, register #5 detects that the items remain in the register. Therefore, in the example in Figure 23, the message "Customer absent, items present" is displayed for register #5, indicating that there is no customer near the register for a predetermined period of time and that the items remain in the register. Additionally, in the example in Figure 23, the movement history information of customer X, who was paying at register #5, indicates that customer X moved in a direction other than the entrance / exit. Therefore, a message is displayed for register #5 indicating that customer X, who was paying at register #5, moved in a direction other than the entrance / exit. Moreover, these conditions—that the customer is absent from the register but the items remain in the register, and that customer X, who was paying at register #5, moved in a direction other than the entrance / exit—indicate a low probability of non-payment. Therefore, a message is displayed for register #5 indicating a low probability of non-payment.

[0242] In the example in Figure 23, during the checkout process, that is, from the start to the end of the checkout, register #6 detects that there is no customer near the register for a predetermined period of time. Furthermore, register #6 detects that there are no items remaining in the register. Therefore, in the example in Figure 23, register #6 displays the message "Customer absent, no items," indicating that there is no customer near the register for a predetermined period of time and no items remain in the register. Additionally, in the example in Figure 23, the movement history information of customer Y, who was paying at register #6, indicates that customer Y moved towards the exit. Therefore, register #6 displays a message indicating that customer Y, who was paying at register #6, moved towards the exit. Moreover, these conditions—that the customer is absent from the register, no items remain in the register, and customer Y, who was paying at register #6, moved towards the exit—indicate a high probability of non-payment. Therefore, register #6 displays a message indicating a high probability of non-payment.

[0243] In the example in Figure 23, during the checkout process, that is, from the start to the end of the checkout, register #2 detects that there is no customer near the register for a predetermined period of time. Furthermore, register #2 detects that there are no items remaining in the register. Therefore, in the example in Figure 23, register #2 displays the message "Customer absent, no items," indicating that there is no customer near the register for a predetermined period of time and no items remain in the register. Additionally, in the example in Figure 23, the movement history information of customer Z, who was paying at register #2, indicates that customer Z moved in a direction other than the entrance / exit. Therefore, register #2 displays a message indicating that customer Z, who was paying at register #2, moved in the direction of the entrance / exit. Moreover, these conditions—that the customer is absent from the register, no items remain in the register, and customer Z, who was paying at register #2, moved in a direction other than the entrance / exit—indicate a low probability of non-payment. Therefore, register #2 displays a message indicating a low probability of non-payment.

[0244] In Figure 23, regarding cash register #2, although no items remained in the register, customer Z, who was paying at cash register #2, moved in a direction other than the exit, it was determined that the possibility of non-payment was low. However, this disclosure is not limited to this. In determining the possibility of non-payment, the presence or absence of items in the register may take precedence over the direction the customer moved. If the presence or absence of items in the register takes precedence, in the example of Figure 23, the fact that the customer was not near the register at cash register #2, no items remained in the register, and customer Z, who was paying at cash register #2, moved in a direction other than the exit, is determined to indicate a high possibility of non-payment.

[0245] In Figure 23, we show examples where messages are displayed for both cases where non-payment is highly likely and cases where non-payment is unlikely. However, while a message stating "There is a high possibility of non-payment" is displayed for cases where non-payment is highly likely, a message stating "There is a low possibility of non-payment" is not required for cases where non-payment is unlikely.

[0246] For example, if a store employee sees the display shown in Figure 22 or Figure 23, they may take immediate action to find the person who committed the shoplifting by rushing to register #6 or by informing other staff members that there is a high probability that shoplifting has occurred.

[0247] The messages shown for cash registers #2, #5, and #6 in Figure 22 or Figure 23 may be displayed regardless of whether the store clerk operates the store clerk monitor or not.

[0248] In the second example of non-payment detection, information about customer behavior is notified to employee monitors, allowing employees to take appropriate action.

[0249] Figure 24 is a flowchart showing a first example of the processing flow for detecting unpaid bills. The processing shown in Figure 24 starts, for example, when the start of accounting is detected. Also, Figure 24 shows the flow executed for one cash register. For example, if the start of accounting is detected for N cash registers (where N is an integer of 1 or more), the flow shown in Figure 24 may be executed for each of the N cash registers.

[0250] The information processing device 10 determines whether or not it has detected the end of accounting (S501).

[0251] If the end of accounting is detected (YES in S501), the flow shown in Figure 24 will terminate.

[0252] If the completion of the transaction has not been detected (NO in S501), the information processing device 10 determines whether or not the customer is absent from the register for a predetermined period of time (S502).

[0253] If the customer is not absent from the register for the specified time (NO in S502), for example, if the customer is present, or if the period during which the customer is absent is less than the specified time, the flow returns to S501.

[0254] If the customer is not at the register for a predetermined period of time (YES in S502), the information processing device 10 determines whether or not there are any items left at the register (S503).

[0255] If there are no items left in the register (NO in S503), the information processing device 10 determines whether the customer who was using the register has moved towards the exit (S504).

[0256] If the customer moves towards the entrance / exit (YES in S504), the information processing device 10 controls the store clerk's monitor to display a warning about unpaid bills (S505). Then the flow shown in Figure 24 ends.

[0257] If there are items remaining in the register (YES in S503), or if the customer is not moving towards the exit (NO in S504), the information processing device 10 controls the employee monitor to display information regarding whether or not there are items remaining in the register and / or the direction the customer is moving (S506). Then the flow in Figure 24 ends.

[0258] Figure 25 is a flowchart illustrating a second example of the processing flow for detecting unpaid transactions. The process shown in Figure 25 is initiated, for example, when the start of accounting is detected. Figure 25 also shows the flow executed for one cash register. For example, if the start of accounting is detected for N cash registers (where N is an integer greater than or equal to 1), the flow shown in Figure 25 may be executed for each of the N cash registers. Note that in Figure 25, processes similar to those shown in Figures 24 and 18 may be given the same numbering and their explanations may be omitted.

[0259] The information processing device 10 determines whether or not it has detected the end of accounting (S501).

[0260] If the end of accounting is detected (YES in S501), the flow shown in Figure 24 will terminate.

[0261] If the completion of accounting has not been detected (NO in S501), the information processing device 10 determines whether or not a scan of the product to be registered (hereinafter referred to as product #k) has been performed (S102).

[0262] If a scan has not been performed (NO in S102), the information processing device 10 determines whether or not it has detected that product #k has moved to the endpoint area (S105).

[0263] If the system does not detect that product #k has moved to the end area (NO in S105), that is, if the customer has not scanned the product and has not moved the product, the information processing device 10 determines whether or not the customer is absent from the register for a predetermined time (S502).

[0264] If the customer is not absent from the register for the specified time (NO in S502), for example, if the customer is present, or if the period during which the customer is absent is less than the specified time, the flow returns to S501.

[0265] If the customer is not at the register for a predetermined period of time (YES in S502), the information processing device 10 determines whether or not there are any items left at the register (S503).

[0266] If there are no items left in the register (NO in S503), the information processing device 10 determines whether the customer who was using the register has moved towards the exit (S504).

[0267] If the customer moves towards the exit (YES in S504), the information processing device 10 controls the store clerk's monitor to display a warning about unpaid bills (S505). Then the flow shown in Figure 25 ends.

[0268] If there are items remaining in the register (YES in S503), or if the customer is not moving towards the exit (NO in S504), the information processing device 10 controls the employee monitor to display information regarding whether or not there are items remaining in the register and / or the direction the customer is moving (S506). Then the flow in Figure 25 ends.

[0269] Cases where a customer leaves items on the register and is not present for the specified time between the start and end of checkout include cases where the customer goes to call a staff member while the register is being operated, cases where the customer goes to the shelves to retrieve items while the register is being operated, and cases where the customer returns items to the shelves while the register is being operated. In such cases, the likelihood of the customer taking the items without paying is low. On the other hand, if a customer is not present at the register for the specified time between the start and end of checkout, and the items are not present at the register, there is a high probability that the customer is attempting to take the items without paying.

[0270] In this embodiment, by detecting the above-mentioned cases and notifying the store staff, the staff can take appropriate action.

[0271] Furthermore, if a customer leaves items on the register and does not remain at the register for the allotted time between the start and end of checkout, and moves in a direction other than the store exit, the customer's actions can be inferred based on the direction of their movement. For example, if the direction of movement is towards the product shelves, it is highly likely that the customer went to retrieve items while the register was being operated, or returned items to the shelves while the register was being operated. Also, if the direction of movement is towards the area where store employees are waiting, it is highly likely that the customer went to call an employee while the register was being operated. If a customer takes such actions, it is unlikely that they will take the items without paying. On the other hand, if a customer does not remain at the register for the allotted time between the start and end of checkout, and there are no items on the register, and the customer does not remain at the register for the allotted time, and moves in the direction of the store exit, it is highly likely that the customer is trying to take the items without paying.

[0272] In this embodiment of non-payment detection, the store clerk can take appropriate action by detecting the above-mentioned cases and notifying them.

[0273] In this embodiment, the non-payment detection detects whether a customer is absent from the register for a predetermined period of time between the start and end of the transaction. If the customer is absent from the register for a predetermined period of time, it detects whether the goods remain in the register, and the detection result is notified to the employee monitor. For example, the employee monitor may be notified if the customer is absent from the register for a predetermined period of time between the start and end of the transaction and the goods remain in the register, or it may be notified if the customer is absent from the register for a predetermined period of time between the start and end of the transaction and the goods do not remain in the register. In other words, the display content on the employee monitor changes depending on whether the goods remain in the register when the customer is absent from the register for a predetermined period of time between the start and end of the transaction. Note that the employee monitor may be notified if the customer is absent from the register for a predetermined period of time and the goods do not remain in the register, while it may not be notified if the customer is absent from the register for a predetermined period of time and the goods remain in the register. In the former case, there is a suspicion that shoplifting has occurred, while in the latter case, it is highly likely that the cashier has temporarily left the register for some reason.

[0274] Furthermore, in the non-payment detection in this embodiment, it is detected whether the customer is absent from the register for a predetermined period of time or longer between the start and end of the transaction. If the customer is absent from the register for a predetermined period of time or longer, it is detected whether the goods remain at the register. The detection result and information on the direction the customer moved are then notified to the employee's monitor.

[0275] <Variation 1 of Non-Payment Detection> The above-mentioned examples of non-payment detection apply regardless of whether the employee operates the employee monitor or not. In other words, regardless of whether the employee operates the employee monitor or not, information will be notified to the employee monitor based on the non-payment detection notification method described above. For example, notification of information based on the non-payment detection notification method described above takes precedence over the employee's operation of the employee monitor. If non-payment is detected, there is a possibility that shoplifting has occurred, and the employee needs to take priority over other tasks, so notification of information based on the non-payment detection notification method takes precedence. In addition to non-payment detection, there may be other actions that take precedence over the employee's operation of the employee monitor. For example, if a person suspected of being a habitual shoplifter is near the register, or if a condition is detected where the probability of shoplifting is higher than a predetermined level, notification of this information to the employee monitor may take precedence over the employee's operation of the employee monitor.

[0276] <Variation 2 for detecting non-payment> When a specific person is making a purchase, the customer monitor may remain displaying that non-payment has been detected, regardless of whether the store clerk performs a predetermined operation via the control panel. Here, the specific person may be a habitual shoplifter, for example. By setting a habitual shoplifter, for example, as a "specific person," the store clerk can focus their vigilance on that "specific person." For example, existing AI image recognition technology may be used to identify whether the person making the purchase is a "specific person."

[0277] <Variation 3 for Non-Payment Detection> In the first and second examples of non-payment detection described above, if there is a high probability of non-payment, the message "There is a high probability of non-payment" will be displayed on the employee monitor. If there is a low probability of payment, the message "There is a low probability of non-payment" does not need to be displayed on the employee monitor.

[0278] <Variation 4 of Non-Payment Detection> In the first and second examples of non-payment detection described above, the display method of the information shown on the employee monitor (e.g., font size and color) can be arbitrarily selected. For example, if there is a high probability of non-payment, the employee may pre-select a display method such as making the font larger or changing the font color compared to when the probability of non-payment is low. This can more strongly alert the employee when there is a high probability of non-payment.

[0279] <Variation 5 of Non-Payment Detection> In the first and second examples of non-payment detection described above, whether or not an item remains at the register is determined based on one of the following detections. That is, the result of determining whether or not an item remains at the register changes depending on the positional relationship between the basket and the item. Note that the basket here may be a shopping basket for items before scanning, or a dedicated shopping basket for items after scanning. Furthermore, the following detections are performed using existing AI image recognition technology. However, these detections are performed when determining non-payment, and are not performed for object detection of items or detection of missed scans. - The item is placed at the register with an item inside a basket. - The item is placed at the register outside of a bag or shopping bag. - The item is placed at the register without being in a basket, bag, shopping bag, etc.

[0280] (Example of the configuration of the information processing device 10) Figure 26 is a block diagram showing an example of the configuration of the information processing device 10. The information processing device 10 shown in Figure 26 has a receiving unit 101, a control unit 102, and a transmitting unit 103. The control unit 102 includes an object detection unit 104, a feature point detection unit 105, a voice detection unit 106, a person detection unit 107, a state determination unit 108, a shoplifting determination unit 109, an unpaid determination unit 110, a display information generation unit 111, a display control unit 112, and a transmission control unit 113. The control unit 102 may be configured by a processor such as a CPU (central processing unit).

[0281] The receiving unit 101 acquires camera images from, for example, the camera 20. For example, if the camera 20 includes an object detection camera 20 and a feature point detection camera 20, the receiving unit 101 acquires an object detection camera image captured by the object detection camera 20 and a feature point detection camera image captured by the feature point detection camera 20. Note that the object detection camera image and the feature point detection camera image may be the same camera image. In other words, the camera 20 may capture an image that can detect feature points on the screen displayed on the display 4 and can also detect products being moved by the customer.

[0282] The receiving unit 101 may also acquire sound from the microphone 30.

[0283] The object detection unit 104 detects objects based on camera images (for example, camera images for object detection). For example, the object detection unit 104 detects from the camera images what kind of goods a customer is holding and / or what kind of goods a customer is moving. The object detection unit 104 outputs object detection information about the detected object to the shoplifting detection unit 109 and the non-payment detection unit 110. The object detection information includes information indicating whether or not the object detection unit 104 has detected an object. If the object detection unit 104 has detected an object, the object detection information may also include information indicating what the detected object is. If the object detection unit 104 has detected the start of movement of an object, the object detection information may also include information indicating the timing of the start of movement of the detected object. Furthermore, the object detection information may include the direction of movement of the detected object, the destination, the position where the movement was completed, etc.

[0284] Furthermore, in the object detection process, the object detection unit 104 may, instead of detecting what the product held by the customer is, detect whether or not the customer is holding an object. Alternatively, instead of detecting what the product the customer is moving is, the object detection unit 104 may detect whether or not the customer is moving an object. In other words, in the object detection process, detection (recognition) of what the object is is not required. In this case, the object detection information may include information indicating whether or not the customer is holding an object, and / or information indicating whether or not the customer is moving an object. Also, the object detection unit 104 may perform the object detection process based on other information instead of camera images (for example, camera images for object detection).

[0285] The feature point detection unit 105 detects feature points in the camera image. For example, the feature point detection unit 105 detects feature points in an image that indicates the processing status of the cash register, corresponding to the display on the display 4, and outputs image feature point information indicating the detected image feature points to the state determination unit 108.

[0286] The voice detection unit 106 detects sounds and other noises that occur when a product is scanned, based on the sound picked up by the microphone 30.

[0287] The person detection unit 107 detects people (e.g., customers) present around the cash register based on camera images (e.g., camera images for object detection). The person detection unit 107 also detects movement history based on camera images, indicating when and in what direction people present around the cash register moved. The person detection unit 107 outputs the detection results to the non-payment determination unit 110 and the display information generation unit 111.

[0288] The state determination unit 108 determines the processing state of the register corresponding to the display on the display 4. For example, the state determination unit 108 determines the processing state of the register corresponding to the display on the display based on image feature point information. Alternatively, the state determination unit 108 may determine that the processing state of the register is the registration processing state if it detects the sound generated when an item is scanned. The processing state of the register to be determined is at least one of the states shown in Figure 3, for example. Note that determination and detection may be interchangeable.

[0289] The shoplifting detection unit 109 determines whether or not a scan error has occurred. Determining whether or not a scan error has occurred corresponds to determining whether or not a "shoplifting act" has occurred. For example, the shoplifting detection unit 109 determines whether or not a scan error has occurred based on the register processing status determined by the status determination unit 108 and the object detection information obtained from the object detection unit 104. For example, if the object detection information indicates that the product has finished moving to the end area and the register processing status indicates that the product is not being registered, the shoplifting detection unit 109 determines that a scan error has occurred. For example, if the object detection information indicates that the product has finished moving to the end area and the register processing status indicates that the product is being registered, the shoplifting detection unit 109 determines that no scan error has occurred.

[0290] The non-payment determination unit 110 determines, based on the detection results from the person detection unit 107 and the processing status of the register from the status determination unit 108, whether or not a customer has been present at the register for a predetermined period of time or longer between the start and end of the transaction. If the non-payment determination unit 110 determines, based on the object detection information, whether or not there are goods present at the register. If there are no goods present at the register, the non-payment determination unit 110 determines, based on the detection results from the person detection unit 107, whether or not the customer's direction of movement is towards the entrance / exit. If the customer's direction of movement is towards the entrance / exit, the non-payment determination unit 110 determines that there is a high probability that payment has not been made. If there are goods present at the register, and / or if the customer's direction of movement is not towards the entrance / exit, the non-payment determination unit 110 determines that there is a low probability that payment has not been made. The non-payment determination unit 110 outputs the determination result and the information used for the determination to the display information generation unit 111.

[0291] The display information generation unit 111 generates information to be displayed on the customer monitor and / or the employee monitor based on the results of the shoplifting detection unit 109 and / or the results of the non-payment detection unit 110. The display information generation unit 111 outputs the information to be displayed on the employee monitor to the transmission control unit 113. The display information generation unit 111 outputs the information to be displayed on the customer monitor to the display control unit 112. The information to be displayed on the employee monitor includes at least one of the following: whether or not shoplifting occurred, an image of the stolen item that was not scanned, whether or not non-payment occurred, and a camera image from a predetermined time ago. The information to be displayed on the customer monitor includes at least one of the following: whether or not shoplifting occurred, an image of the stolen item that was not scanned, whether or not non-payment occurred, and a camera image from a predetermined time ago.

[0292] The display control unit 112 controls the display on the display 40 installed in the cash register. For example, the display control unit 112 causes the display 40 to display information according to the display information or instructions generated by the display information generation unit 111 to be displayed on the customer monitor.

[0293] The transmission control unit 113 generates transmission information from the display information shown on the employee monitor and transmits the transmission information to the store terminal 200 via the transmission unit 103. The transmission information may be transmitted to terminals owned by store staff, security guards, etc., or to display devices that store staff, security guards, etc., can check. The transmission information may also be transmitted to display devices located outside the store, such as display devices for remotely monitoring shoplifting.

[0294] (Example of configuration of store terminal 200) Figure 27 is a block diagram showing an example of the configuration of a store terminal 200. The store terminal 200 shown in Figure 27 has a receiving unit 201, a control unit 202, a transmitting unit 203, a display 204, and an operation unit 205. The control unit 202 includes an information classification unit 211, a display control unit 212, a parameter setting unit 213, and a transmission control unit 214. The control unit 202 may be configured by a processor such as a CPU (central processing unit).

[0295] The receiving unit 201 acquires transmission information from the information processing device 10. The transmission information includes display information to be displayed on the display 204.

[0296] The information classification unit 211 classifies the transmitted information into categories such as information indicating the processing status of each register, images of items that were not scanned, and camera images.

[0297] The parameter setting unit 213 sets parameters for the information to be displayed on the display 204 based on the store clerk's operation information obtained via the operation unit 205. For example, the parameter setting unit 213 sets parameters such as which image to display.

[0298] The display control unit 212 controls the display on the display 204. For example, based on a parameter indicating which set information to display, the display control unit 212 controls at least one of the following on the display 204: updating the processing status of each register, displaying images of items that were not scanned, and displaying camera images.

[0299] The operating unit 205 may be, for example, a touch panel superimposed on the display 204, or it may be a device such as a keypad or mouse.

[0300] When the transmission control unit 214 receives operation information from the operation unit 205 requesting information to be displayed, it generates transmission information to request the information to be displayed from the information processing device 10 based on the operation information. For example, the transmission information includes at least one of the following: information indicating the register selected by the operation unit 205, information indicating the number of images of products that were not scanned and are requested, and information indicating the time of the camera image to be displayed. The transmission control unit 214 transmits the transmission information to the information processing device 10 via the transmission unit 203.

[0301] In this embodiment, an example is shown in which an information processing device 10 provided at each self-checkout counter performs object detection processing and shoplifting detection, but this disclosure is not limited to this. At least one of the object detection processing and shoplifting detection may be performed by other devices such as a store terminal 200 or a server installed in the store. For example, if the object detection processing is performed by another device, the information processing device 10 transmits camera images, etc., to the other device (e.g., a store terminal 200, a server installed in the store, etc.) and obtains the results of the object detection processing from the other device.

[0302] Furthermore, in the above embodiment, the information processing device 10 detected the movement of the product using camera images. However, the movement of the product may be detected by other means. For example, the information processing device 10 may detect the movement of the product based on changes in weight detected by weight sensors provided on product stands A1a and A1b. Alternatively, for example, if the product is fitted with a wireless tag such as an RFID, the information processing device 10 may detect the movement of the product by detecting the movement of the wireless tag. In addition, the information processing device 10 may detect the movement of the product by using a combination of camera images for object detection and detection results from weight sensors or wireless tags.

[0303] Furthermore, in the above embodiment, the information processing device 10 determined the processing status of the cash register corresponding to the display on the display 4 by detecting feature points of the image. However, the information processing device 10 may also determine the processing status of the cash register using other methods, such as recognizing characters displayed on the display 4 or using pattern matching of the entire image displayed on the display 4.

[0304] Furthermore, in this disclosure, shoplifting detection does not necessarily have to be a function of the cash register. For example, a server may perform shoplifting detection based on camera images sent from the cash register.

[0305] In this disclosure, the camera is assumed to be externally attached to the cash register, but it may also be built into the cash register.

[0306] In this embodiment, the scan determination is made from the register screen or sound, but information indicating that a scan has been performed may also be obtained directly from the register.

[0307] In this embodiment, the "image of a product that was not scanned" may be, for example, "a camera image of the area around the self-checkout counter (which also shows the customer and the scanned product), where the product is being moved while being held in the hand, and the camera image was detected at that moment." Whether or not the product is being moved while being held in the hand may be determined by the history of changes in the position where the product was detected, and / or whether or not a hand was detected. Furthermore, the information on whether or not the product is being held in the hand may also be used for shoplifting detection, for example, to determine that it is not shoplifting while the product is being held in the hand. However, if the information on whether or not the product is being moved while being held in the hand is not used for shoplifting detection, such as when shoplifting detection is performed using a weight sensor, then "being moved while being held in the hand" does not need to be included as a condition for the "image of a product that was not scanned."

[0308] In addition, the "...part" in the above embodiment may be a "...circuit," a "...device," a "...unit," or a "...module."

[0309] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0310] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0311] The integrated circuit implementation method is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, a Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0312] Furthermore, if advancements in semiconductor technology or other derived technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, it would be possible to use those technologies to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0313] This disclosure is applicable to all types of devices, systems, and equipment with communication capabilities (collectively referred to as communication equipment). Non-exclusive examples of communication equipment include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth and telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned equipment.

[0314] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting fixtures, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0315] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0316] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0317] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0318] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0319] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above.

[0320] The disclosures in the specification and abstract included in the Japanese application 2024-232236, filed on December 27, 2024, and the disclosures in the specification and abstract included in the Japanese application 2025-40232, filed on March 13, 2025, are all incorporated herein by reference.

[0321] This disclosure is useful for detecting fraudulent activity in self-checkout systems.

[0322] 1.10 Information processing device 2. Scanner 3.20 Camera 30 Microphone 4.40 Display A1a, A1b Product stand 101 Receiving unit 102 Control unit 103 Transmitting unit 200 Store terminal

Claims

1. A monitoring system comprising: a store clerk monitor that displays the status of product payment at each of the multiple self-checkout registers; and a control unit that controls the display on the store clerk monitor, wherein the store clerk monitor displays that a customer has left the self-checkout register if the customer making the payment for the product has been away from the self-checkout register for a predetermined period of time or longer.

2. The monitoring system according to claim 1, wherein the employee monitor changes the content displayed on it depending on whether the product is present in the vicinity of the self-checkout register when the customer has been away from the self-checkout register for a predetermined period of time or longer.

3. The monitoring system according to claim 2, wherein the employee monitor displays a warning if the customer has been away from the self-checkout for a predetermined amount of time and the product is not present in the vicinity of the self-checkout, indicating that the customer may not have paid for the product.

4. The monitoring system according to claim 3, wherein the employee monitor does not display the warning if the customer is away from the self-checkout for a predetermined amount of time and the product is located near the self-checkout.

5. The monitoring system according to claim 2, wherein the employee monitor displays information indicating whether or not the product is present in the vicinity of the self-checkout register when the customer has been away from the self-checkout register for a predetermined period of time or longer.

6. The monitoring system according to claim 2, wherein the employee monitor changes the content displayed on it according to the direction of the customer's movement if the customer is away from the self-checkout register for a predetermined amount of time or longer and the product is not present in the vicinity of the self-checkout register.

7. The monitoring system according to claim 6, wherein the employee monitor displays a warning that the customer may not have paid for the goods if the customer has been away from the self-checkout for a predetermined amount of time, the goods are not located near the self-checkout, and the customer is moving towards the exit of the store.

8. The monitoring system according to claim 6, wherein the employee monitor displays information indicating whether or not the product is not present in the vicinity of the self-checkout register and information indicating the direction of the customer's movement when the customer has been away from the self-checkout register for a predetermined period of time or longer.

9. The monitoring system according to claim 1, wherein the employee monitor displays an image of a product for which the customer did not register product information at the selected self-checkout register, in response to the operation of selecting one of the plurality of self-checkout registers.

10. The monitoring system according to claim 1, wherein the employee monitor displays an image of the customer registering product information at the selected self-checkout register in response to the employee's operation to select one of the plurality of self-checkout registers.

11. The monitoring system according to claim 1, wherein the employee monitor displays either an image of a product for which the customer did not register product information at the selected self-checkout register, or an image of the customer registering product information at the selected self-checkout register, in response to an operation to select one of the plurality of self-checkout registers and an operation to select information to display.

12. The monitoring system according to claim 1, further comprising a customer monitor that displays information relating to the accounting in the vicinity of the self-checkout register, wherein the customer monitor does not display that the customer is away from the self-checkout register if the customer is away from the self-checkout register for a predetermined period of time or longer.

13. The monitoring system according to claim 12, wherein the customer monitor displays images of products for which the customer did not register product information at the self-checkout counter corresponding to the customer monitor.

14. The monitoring system according to claim 13, wherein the customer monitor maintains the display of the image according to the settings and cancels the display of the image according to a change in the settings.

15. A display method comprising: a monitor for store staff displaying the status of product payment at each of several self-checkout registers; and the monitor for store staff displaying that a customer is away from one or more of the self-checkout registers if the customer is away from the self-checkout register for a predetermined period of time or longer.

Images