Information processing device and control method

Abstract

This information processing device comprises: a reception unit that acquires an image capturing, in a store, the actions of a customer of moving an article placed in a first region to a second region, registering the article with a registration device, and moving the article from the second region to a third region; and a control unit that performs a process to detect, on the basis of the image, whether or not the customer registered a first article in the second region, and that, when it is detected that the customer did not register the first article in the second region, performs control to cause a first notification method for notifying the customer of the detection result and a second notification method for notifying staff of the store of the detection result to be different from each other.

Description

Information Processing Apparatus and Control Method

[0001] The present disclosure relates to an information processing apparatus and a control method.

[0002] In recent years, due to the reduction of labor costs for store employees in retail stores and the like, the popularity of self-checkout has been rapidly advancing. In self-checkout, since the customer himself / herself scans the barcode of the product, it is important to take measures against illegal acts such as shoplifting where the product is taken away without paying for it.

[0003] For example, in Patent Document 1, when it is determined that a product has been taken out of the shopping basket, image data captured by a recognition camera is acquired and analyzed to determine whether the product has been put into a storage bag. When it is determined that the product has been put into the storage bag, it is determined whether the barcode of the product has been read and scanned by a barcode scanner between the shopping basket and the conducting wire of the storage bag. When it is determined that the product has not been scanned between the shopping basket and the conducting wire of the storage bag, an error warning is notified.

[0004] Japanese Patent Application Laid-Open No. 2011-054038

[0005] However, there is room for consideration regarding the method of notifying the determination result such as whether an illegal act has occurred.

[0006] The non-limiting embodiments of the present disclosure contribute to providing an information processing apparatus and a control method that can appropriately notify the determination result such as whether an illegal act has occurred.

[0007] An information processing apparatus according to an embodiment of the present disclosure includes a receiving unit that acquires an image of an operation until a customer in a store moves an article placed in a first area to a second area and registers the article with a registration device, and then moves the article from the second area to a third area. A control unit that executes a detection process based on the image to detect whether the customer is registering a first article in the second area, and when it is detected that the customer is not registering the first article in the second area, performs control to differ a first notification method for notifying the customer of the detection result and a second notification method for notifying the store staff of the detection result.

[0008] In one embodiment of the present disclosure, the control method involves an information processing device acquiring images of the actions taken by a customer in a store, from moving an item placed in a first area to a second area and registering the item using a registration device, to moving the item from the second area to a third area, and based on the images, performing a detection process to determine whether the customer has registered the first item in the second area. If it is detected that the customer has not registered the first item in the second area, the device performs control to differentiate between a first notification method for notifying the customer of the detection result and a second notification method for notifying the store staff of the detection result.

[0009] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0010] According to one embodiment of this disclosure, the results of the determination of whether or not fraudulent activity has occurred can be appropriately notified.

[0011] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features.

[0012] Diagram showing an example of a self-checkout system configuration Diagram showing an example of area setting Diagram showing an example of product movement in an area setting Diagram showing an example of chain state transition Diagram showing an example of chain lifecycle Diagram showing an example of the processing flow when performing shoplifting possibility determination Diagram showing an example of a notification method Diagram showing an example of the processing flow for countermeasures against false detection in scene detection Diagram showing an example of the processing flow for disabling movement detection Block diagram showing an example of information processing device configuration

[0013] The embodiments of this disclosure will be described in detail below, with reference to the drawings as appropriate. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding for those skilled in the art.

[0014] The attached drawings and the following description are provided to enable a person skilled in the art to fully understand this disclosure, and are not intended to limit the subject matter described in the claims.

[0015] <Embodiment> (System Configuration) Figure 1 is a diagram showing an example of the configuration of a self-checkout system. As shown in Figure 1, the self-checkout system includes an information processing device 1, a scanner 2, a camera 3, and a display 4. Figure 1 also shows an information processing device 10, a camera 20, a microphone 30, and a display 40 according to this embodiment. In addition to the self-checkout system, Figure 1 also shows product display stands A1a and A1b. Hereinafter, the self-checkout system may be simply referred to as "self-checkout," "cash register," or "POS (Point of Sales) register."

[0016] Product display stand A1a is a stand on which products are placed before being scanned by scanner 2. Products placed on product display stand A1a before scanning may include shopping baskets or carts from the store containing the products before scanning. In the following description, "product" may refer to "articles" or "objects" that are different from the products actually sold in the store. In the following description, "product" may be replaced with "articles" or "objects" as appropriate.

[0017] Product display stand A1b is a stand on which products that have been scanned by scanner 2 are placed. Scanned products placed on product display stand A1b may include the customer's shopping basket or other items in which the scanned products are placed.

[0018] The information processing device 1 is, for example, a computer such as a personal computer or a server. In Figure 1, the information processing device 1 is placed on the product display stand A1a, but it may also be placed in the store's office. The information processing device 1 performs, for example, accounting processing for products.

[0019] Scanner 2 is a fixed-type scanner that is attached to the product display stand A1a. Although not shown in Figure 1, a handheld scanner may also be provided.

[0020] Scanner 2 is connected to information processing device 1. Scanner 2 scans the product code attached to the product. Scanner 2 transmits the scanned product code to information processing device 1. The product code is a barcode, such as a JAN (Japanese Article Number) code. Scanner 2 may also be called a barcode reader or reader. The area where Scanner 2 scans the product code attached to the product may be called the "scan area". The customer scans the product code by bringing the product close to the scan area and passing it through. Scanning the product code attached to the product may also be simply called scanning the product. Furthermore, if a handheld scanner is used as Scanner 2, the position of the scan area does not have to be fixed and may change according to the range of motion of the handheld scanner.

[0021] Camera 3 is connected to the information processing device 1. Camera 3 transmits image data of the captured image to the information processing device 1.

[0022] Camera 3 is installed to photograph customers. However, camera 3 is not required.

[0023] The display 4 is connected to the information processing device 1. The display 4 displays, for example, product information of an item scanned by a customer using the scanner 2. The product information may include, for example, the name, price, and quantity of the product.

[0024] Display 4 may have a touch panel on its surface. The touch panel accepts customer input. For example, a customer can input information about products that do not have product codes, such as vegetables or fruits, via the touch panel and register them as purchased items. For example, if a customer purchases two apples, which do not have product codes such as barcodes, they can input the product name "apple" and the quantity "2" via the touch panel and register them as purchased items. Note that the device that accepts customer input is not limited to a touch panel. The device that accepts customer input may be, for example, a key input device separate from Display 4. Alternatively, customer input may be accepted using a smartphone or mobile device owned by the customer.

[0025] In the following, the actions of scanning product codes attached to products, and the actions of manually entering information for products that do not have product codes, may be collectively referred to as "product information registration" or "product registration."

[0026] Furthermore, in the following, the series of processes at the register from when a customer begins registering product information until they pay for the registered products will be referred to as "accounting processing." The process of registering product information will be referred to as "registration processing," and the process of paying for the registered products will be referred to as "payment processing."

[0027] Display 4 displays, for example, information corresponding to the processing status of the cash register. For example, the processing status of the cash register may include "waiting for accounting processing," "waiting for registration processing," "registration processing," and "waiting for payment."

[0028] The "checkout processing waiting" state indicates the cash register is waiting to process an transaction before a customer operates the register to complete the transaction. The "registration processing waiting" state refers to the state where a customer is operating the register to complete the transaction, and the register is waiting for the customer to register the product information. The "registration processing" state refers to the state within a specific time period from the moment the product information is registered. The "payment waiting" state refers to the state where the customer has completed registering the information for the products they wish to purchase, and the register is waiting to process the payment.

[0029] The information processing device 1 performs accounting processing for the self-checkout system. For example, the information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (e.g., the results of product information input) via the touch panel on the surface of the display 4, and performs calculations for the total price of the products registered by the customer and payment processing for the amount paid by the customer.

[0030] The information processing device 1 acquires the results of scanning product codes by the scanner 2 and customer operation information (for example, the result of inputting product information) via the touch panel on the surface of the display 4, and performs display control to display product information of the products registered by the customer on the display 4. In addition, the information processing device 1 performs display control to display information corresponding to the processing status of the register on the display 4 based on the customer operation information via the touch panel on the surface of the display 4.

[0031] In this implementation, the information processing device 10, camera 20, microphone 30, and display 40 are installed externally later on the self-checkout system.

[0032] Camera 20 is positioned to photograph the area around where the customer scans the product with scanner 2, and to photograph at least a portion of the display area of ​​display 4. For example, camera 20 is positioned above display 4. The field of view of camera 20 is set to include the area between product stands A1a and A1b, that is, the area around where the customer scans the product with scanner 2, and also include at least a portion of the display area of ​​display 4.

[0033] However, the camera 20 does not need to photograph the area where the customer scans the product with the scanner 2. In this case, the camera 20 is positioned to photograph at least a portion of the display area of ​​the display 4.

[0034] Note that the camera 20 is not limited to one. Multiple cameras 20 may be installed. For example, a camera 20 may be installed at a position that photographs the area around where the customer scans the product with the scanner 2, and at a position that photographs at least a portion of the display area of ​​the display 4. When cameras 20 are installed at these two positions, the camera 20 installed at the position that photographs the area around where the customer scans the product with the scanner 2 may be called an object detection camera 20, and the camera 20 installed at the position that photographs at least a portion of the display area of ​​the display 4 may be called a feature point detection camera 20. The camera image captured by the object detection camera 20 may be called an object detection camera image, and the camera image captured by the feature point detection camera 20 may be called a feature point detection camera image.

[0035] In addition, multiple cameras 20 may capture at least a portion of the display area of ​​the display 4, or multiple cameras 20 may capture the area around where the customer scans the product with the scanner 2.

[0036] An example of a camera's field of view will be discussed later.

[0037] The information processing device 10 is, for example, a computer such as a personal computer or a server. The information processing device 10 may be located in the office of the store. The information processing device 10 is connected to the camera 20 by wire or wireless and acquires camera images taken by the camera 20. The camera images may be still images, multiple still images in a sequence over time, or moving images.

[0038] Furthermore, the information processing device 10 may acquire sound from a microphone 30 installed around the self-checkout system. The microphone 30 acquires sound emitted from the self-checkout system and outputs the acquired sound to the information processing device 10. The sound emitted from the self-checkout system includes, for example, sounds corresponding to the processing status of the register, or sounds corresponding to the transition of the processing status of the register.

[0039] The information processing device 10 performs a determination process based on the acquired camera images and / or audio to determine, for example, whether or not an illegal act (shoplifting) has occurred. The determination of whether or not shoplifting has occurred is sometimes referred to as shoplifting detection. Camera images and / or audio are examples of sensing results of information output by a self-checkout system.

[0040] Note that "determination" and "detection" are interchangeable. For example, the determination of whether or not shoplifting has occurred may be called the detection of whether or not shoplifting has occurred, or shoplifting detection.

[0041] For example, the information processing device 10 performs object detection processing based on the acquired camera image. The information processing device 10 determines the presence of an object and whether or not the object is moving through the object detection processing. Note that the object detection processing does not have to be based on the camera image. The information processing device 10 also determines the status of the cash register based on the acquired camera image and / or audio. Then, the information processing device 10 determines whether or not shoplifting has occurred based on the results of the object detection processing and the status of the cash register. Note that object detection may also be called product detection. Also, the determination of the status of the cash register may be called "scene detection" based on the display area of ​​the display 4. Note that object detection and product detection will be described later.

[0042] The information processing device 10 is connected to the display 40 and controls the display on the display 40.

[0043] The display 40 performs display based on the determination result in the information processing apparatus 10 under the control of the information processing apparatus 10. For example, the display 40 notifies customers of information regarding fraudulent acts. The notification of information regarding fraudulent acts includes, for example, the notification of a message indicating that a scan omission has occurred, the notification of a message indicating that there may be a scan omission, the notification of a cautionary message to prevent a scan omission, etc. The notification of information regarding fraudulent acts may be replaced with a warning regarding fraudulent acts. Here, the fraudulent act may correspond to shoplifting. Note that the shoplifting here may correspond to the act of moving a product with a scan omission to the product stand A1b without scanning the product (an example of registration).

[0044] In FIG. 1, an example is shown in which the information processing apparatus 10 is provided separately from the information processing apparatus 1 of the self-checkout system, but the present disclosure is not limited to this. The processing of the information processing apparatus 10 according to the present embodiment may be executed in the information processing apparatus 1 of the self-checkout system. Further, the information processing apparatus 1 of the self-checkout system may be connected to an external server by wire or wirelessly, and in the external server, the processing of the information processing apparatus 10 according to the present embodiment and / or a part of the processing of the information processing apparatus 1 may be executed.

[0045] Also, in FIG. 1, an example is shown in which the camera 20, the microphone 30, and the display 40 are externally installed in the self-checkout system later, but at least one of the camera 20, the microphone 30, and the display 40 may be built into the self-checkout system.

[0046] There are a plurality of self-checkouts as shown in FIG. 1 in the store. The plurality of self-checkouts communicate with a terminal for store employees provided in the store and exchange information.

[0047] (Determination Algorithm) Here, an example of a method for the information processing apparatus 10 to determine whether or not a shoplifting act has occurred using information obtained from the camera 20 and / or the microphone 30 connected to the information processing apparatus 10 without cooperation with the self-checkout system will be described. The determination of whether or not a shoplifting act has occurred is hereinafter referred to as shoplifting determination. The occurrence of a shoplifting act corresponds to the occurrence of a scanning omission of a product.

[0048] (Use of Camera Images) As illustrated in FIG. 1, the camera 20 is provided above the cash register. The camera 20 is installed at a position where it can photograph the self-checkout monitor (e.g., the display 4 in FIG. 1) at an elevation angle of about 30 degrees substantially vertically from above the cash register downward.

[0049] It is assumed that the camera 20 outputs camera images of 5 FPS (frames per second) or more to the information processing apparatus 10. For example, the camera 20 outputs camera images of 10 FPS to the information processing apparatus 10.

[0050] The information processing apparatus 10 executes two types of detection processes called "product detection" and "scene detection" from the camera images using an object detection engine. In product detection, it is detected how the products purchased by the self-checkout user (e.g., customer) are moved. In scene detection, the cash register operations performed by the self-checkout user are detected from the content displayed on the self-checkout display 4. Note that since product detection may also detect objects other than products sold in a store or the like (e.g., the customer's personal belongings), product detection may be referred to as object detection. Product detection and scene detection will be described later.

[0051] (Use of microphone audio) Microphone 30 is installed near the speaker of the self-checkout machine. Microphone 30 acquires sounds such as those emitted when the self-checkout machine reads the barcode of a product and outputs them to the information processing device 10. The information processing device 10 detects the sound emitted when the self-checkout machine reads the barcode of a product from the sound acquired from the microphone. The sound emitted when the self-checkout machine reads the barcode of a product is called the scan sound. The detection of the scan sound is called scan sound detection. Note that the sound acquired by the microphone is not limited to the scan sound. For example, sounds indicating the operation of the self-checkout machine or sounds indicating the processing status of the self-checkout machine may be acquired by the microphone and detected by the information processing device 10. Scan sound detection will be described later.

[0052] (Definition of camera image area) In the product detection process described above, an area is defined for the camera image taken of the cash register and its surroundings.

[0053] The following is an example of setting an area in a camera image.

[0054] Figures 2A and 2B show examples of area settings. Figure 2A shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the starting area and ending area set within the shooting range. Figure 2B also shows the shooting range from which camera 20 photographs the self-checkout system illustrated in Figure 1 from above, as well as examples of the personal belongings area and dead zone set within the shooting range.

[0055] The starting area is the area where the product before scanning and the shopping basket containing the product are expected to be placed. The starting area corresponds to the area where the movement of the product before scanning begins. The starting area includes at least the area of ​​product stand A1a. The starting area is set within the range of the register, including product stand A1a, but may include a margin beyond the range of the register. Furthermore, a wider area than product stand A1a may be included in the starting area. For example, in addition to the range of product stand A1a, the starting area may include the range of product stand A1a extended toward the customer (for example, in the direction of arrow G in Figure 2A). Also, a part of product stand A1b may be included in the starting area.

[0056] The movement of an item may be determined to have started when the customer picks up the item located in the starting area. Alternatively, the movement of an item may be determined to have started when the position or size of the item in the starting area changes in the camera image.

[0057] The endpoint area is the area where the shopping bags, reusable bags, baskets, etc., that will hold scanned items are expected to be placed. The endpoint area corresponds to the area where the movement of items ends. The endpoint area includes at least the area of ​​product stand A1b. The endpoint area is set within the range of the register, including product stand A1b, but may include a margin beyond the register's range. Furthermore, the endpoint area may include a wider area than product stand A1b. For example, in addition to the range of product stand A1b, the endpoint area may include an area that extends the range of product stand A1b toward the customer (for example, in the direction of arrow H in Figure 2A). Also, a part of product stand A1a may be included in the endpoint area.

[0058] The system may determine that the product has completed its movement to the endpoint area when the product held in the customer's hand moves to the endpoint area and the position and size of the product in the camera image do not change. Alternatively, the system may determine that the product has completed its movement to the endpoint area when the product held in the customer's hand moves to the endpoint area and the customer's hand is detected to have released the product.

[0059] The personal belongings area is the area where a cashier (e.g., a customer) holding personal belongings is expected to be visible, and / or where a customer's personal belongings are expected to be placed. The personal belongings area is set outside the range of the cashier in the camera image. The personal belongings area corresponds to the area where personal belongings that could be mistakenly identified as merchandise (e.g., a customer's wallet, smartphone, etc.) first appear. Note that the personal belongings area does not have to be set. However, at least a part of the range of the cashier in the camera image may be included in the personal belongings area. For example, part or all of the endpoint area described above may be included in the personal belongings area. Also, part of the starting area described above may be included in the personal belongings area. Note that the personal belongings area may overlap with the starting and / or endpoint areas described above. Here, when the personal belongings area and the starting and / or endpoint areas described above overlap, the priority of which one to use may be arbitrarily set. For example, it may be arbitrarily set whether the area where the personal belongings area and the starting area overlap is treated as the personal belongings area or as the starting area. In the example shown in Figure 2B, the personal belongings area may take precedence.

[0060] The dead zone is an area where it is not expected that a cashier operating a register with personal belongings (e.g., a customer) will be captured in the image, and / or where it is not expected that a customer's personal belongings will be placed. The dead zone is set outside the range of the register in the camera image. Because the dead zone may include images of adjacent registers or customers or store staff moving around, the dead zone corresponds to an area that can become noise. Items, people, etc. that appear in the dead zone are excluded from image processing such as product detection. Note that the dead zone does not have to be set.

[0061] Each area may be configured manually based on the camera image obtained when a camera 20 is installed at the cash register, the field of view of the camera 20 is set, etc.

[0062] Although not shown in Figures 2A and 2B, the area scanned by scanner 2 may be set as the scan area.

[0063] Furthermore, areas not shown in Figures 2A and 2B may be set. Also, there may be overlapping areas between the set areas. If there is an overlapping area between two areas, it may be dynamically changed which of the two areas the overlapping area is treated as. For example, an overlapping area that is treated as a personal belongings area at the start of product movement may be treated as a starting area or an ending area at other times.

[0064] The following explains an example of how to move goods when an area is defined as described above.

[0065] Figure 3 shows an example of product movement in relation to area settings. Figure 3 shows an example of a customer moving products in the self-checkout system shown in Figure 1.

[0066] As shown in Figure 3, when a customer scans product #x, the customer moves product #x from the starting area. Then, the customer scans product #x by bringing it close to scanner 2, and moves the scanned product #x to the ending area. The direction in which a product moves from the starting area to the ending area is called the "forward direction." Conversely, the direction in which a product moves from the ending area to the starting area is called the "reverse direction."

[0067] (Product Detection) The product detection process performed by the information processing device 10 is subject to the following preconditions: • Product detection uses a learning model that detects products held in a person's hand. • Product detection targets only one type of product (e.g., a label). • Product detection limits a person to only one product at a time. However, these preconditions are merely examples. At least one of these preconditions may not be applied to the product detection process in this disclosure.

[0068] The information processing device 10 inputs the acquired camera image to the object detection engine and evaluates the circumscribed rectangle coordinates and score value, which are the detection results of the object detection engine. For example, if the score value is greater than or equal to a threshold, the information processing device 10 identifies the circumscribed rectangle corresponding to that score value as a product. If the score value is less than the threshold, the information processing device 10 ignores the circumscribed rectangle corresponding to that score value. If the information processing device 10 detects multiple circumscribed rectangles, it adopts the circumscribed rectangle with the largest score value. If the adopted score value is greater than or equal to a threshold, it identifies the circumscribed rectangle corresponding to that score value as a product, and ignores the other circumscribed rectangles that are not adopted.

[0069] The information processing device 10 treats the center coordinates of the bounding rectangle as the coordinates of the detected product. If no bounding rectangles are detected, the information processing device 10 treats it as no product detected.

[0070] (Scene Detection) The scene detection process performed by the information processing device 10 is subject to the following preconditions: - Scene detection uses a learning model to identify the display on the self-checkout display 4. - Since the display on the display 4 differs depending on the model of the register, learning is required for each model of register. However, the two preconditions described above are just examples. At least one of these preconditions may not be applied to the scene detection process in this disclosure.

[0071] Labels such as "Checkout Start Screen," "Checkout End Screen," "No Barcode Item Screen," and "Item Selection Screen" are defined for the display of the target display 4. The Checkout Start Screen corresponds to a screen indicating that "from this screen onward, the settlement process for each item will be carried out." The Checkout End Screen corresponds to a screen indicating that "the settlement process for each item has been completed and payment processing will be carried out." The No Barcode Item Screen corresponds to a screen indicating that "items without barcodes will be identified by monitor operation." The Item Selection Screen corresponds to a screen indicating that "purchased items have been selected by barcode scanning or monitor operation."

[0072] Furthermore, given the characteristics of the display 4 screen, it can be assumed that multiple screens will not appear.

[0073] (Scene Detection Method) The information processing device 10 inputs the acquired camera image to the object detection engine and evaluates the label and score value, which are the detection results of the object detection engine. If the score value is above a threshold, the information processing device 10 determines that the label corresponding to the score value corresponds to the screen of the display 4 included in the camera image. If the score value is below the threshold, the information processing device 10 ignores the label corresponding to the score value. If the information processing device 10 detects multiple labels as candidates, it adopts the label with the larger score value. If the adopted score value is above a threshold, the information processing device 10 determines that the label corresponding to the score value corresponds to the screen of the display 4 included in the camera image, and ignores the other labels that are not adopted. If no labels are detected, the information processing device 10 determines that a screen other than the screen corresponding to a label is being displayed.

[0074] (Scan Sound Detection) The scan sound processing performed by the information processing device 10 is subject to the following preconditions: - The frequency of the scan sound is uniquely determined for each register. - The frequency of the scan sound can be set in advance. - The scan sound is emitted for 50 to 100 msec. - The difference in volume can be sufficiently distinguished if there is a distance of about 1 m between adjacent registers. In other words, whether the scan sound is emitted from the register to be detected or from a register in the vicinity of the register to be detected is distinguished by the difference in volume, etc. However, these preconditions are just examples. At least one of these preconditions does not have to be applied in the scan sound detection processing in this disclosure.

[0075] (Scan Sound Detection Method) In the scan sound detection method of the information processing device 10, the length of the audio buffer is set to a length that allows for volume measurement by dividing the duration of the scan sound into 5 to 10 parts. The audio buffer is then subjected to a Fast Fourier Transform to set only the volume of the pre-set frequency band as the measurement target. If the volume is below the threshold, it is determined that no scan sound was emitted. The peak time of the volume between the point where the volume exceeds the threshold and then falls below the threshold again is recorded as the scan time.

[0076] (Chain) Next, the chain used in this embodiment will be described. A chain is a unit of management for the movement of goods (e.g., detected goods) held in the hands of a user at a cash register, when the information processing device 10 manages the movement of goods (e.g., detected goods). A chain is generated when a goods are detected after a period of not being detected. The following information is managed in the chain. Note that the coordinates managed in the chain are the coordinates at which the goods corresponding to the chain were detected. - Chain ID - Start and end times of the chain - Start and end coordinates of the chain - All times when goods were detected - All coordinates at which goods were detected - Distance between the position at which goods were detected at a certain point in time T and the position at which goods were detected before point in time T - All area information where goods were detected - Current management position of the chain

[0077] Area information is a list of boolean values ​​indicating whether the coordinates of a product obtained through product detection are included within the area definitions shown in Figures 2A and 2B. For example, area information indicates at least one of the following: "whether it is within the starting area," "whether it is within the ending area," and "whether it is within the personal property area."

[0078] The current management location of a chain indicates the current area for managing the movement of goods. Specifically, it indicates one of the following: • Initial area: The chain does not belong to any defined area. • Personal area: The chain is in the personal area. • Starting area: The chain is in the starting area. • Ending area: The chain is in the ending area.

[0079] If the initial detection location is within the personal belongings area, the personal belongings area will be set as the default value. If the initial detection location is outside the personal belongings area, the default area will be set as the default value.

[0080] (Chain Status) When an item is detected, the chain's management information is updated. For example, at least one of the following is updated: the latest time of item detection, the latest coordinates of item detection, the latest area information of item detection, and the current management location of the chain.

[0081] Figure 4 shows an example of a chain state transition. For example, the information processing device 10 resets the state of each chain according to the state transition matrix in Figure 4.

[0082] For example, if the current management location of a chain is the "initial area," and the boolean value of the latest detection area where a product was detected is the starting area = true, then the current management location of that chain will be updated from the "initial area" to the "starting area." Note that the boolean value of the latest detection area where a product was detected being the starting area = true corresponds to the case where the latest detection area where a product was detected is the starting area.

[0083] In the example shown in Figure 4, if the most recent detection area where an item is detected is within the "personal belongings area," this is not considered and can be ignored. In other words, if the detection location is within the personal belongings area for the second time or later, rather than the first time, the detection result is ignored.

[0084] (Chain lifecycle) The information processing device 10 manages two chains, an active chain and a pending chain, as an example.

[0085] Figure 5 shows an example of a chain's lifecycle. The generated chain follows the lifecycle (living state) illustrated in Figure 5. Figure 5 shows the "non-existence" state, which corresponds to the state without a chain, the "existence state," which is the state in which a chain exists, the "holding state," and examples of transitions between these states.

[0086] As shown in Figure 5, if a product is detected while the chain is absent (not present), the chain transitions to the "present state".

[0087] A chain transitions from an existing state to a pending state when any of the following pending conditions are met.

[0088] The "pending state" refers to a grace period before the chain is cut, and is a state established to prevent accidental cutting due to false detection of objects, etc. A chain in the pending state will return to the existing state when the chain revival conditions described later are met.

[0089] The following explains the three conditions for holding the application.

[0090] Hold Condition 1: No products are detected for a certain period of time or longer. The threshold for the certain period in Hold Condition 1 is, for example, N2 seconds (where N2 is a real number greater than or equal to 0). When Hold Condition 1 is met, there are no chains in the existing state, and only chains in the hold state exist.

[0091] **Holding Condition 2:** Product detection at a point beyond a certain distance. The threshold indicating the certain distance in Holding Condition 2 is, for example, N1 or N1(G2S) (where N1 or N1(G2S) is a real number greater than or equal to 0). Holding Condition 2 is a condition set up, for example, to treat the product initially held and the other product held afterward as separate chains when a customer puts down the product they are holding and picks up another product. Note that the certain distance in Holding Condition 2 varies depending on the relationship between the area where the object was detected immediately before and the area where the object is currently detected.

[0092] Here, N1 used in the holding condition 2 and N1(G2S) are different threshold values.

[0093] N1 (G2S) corresponds to the threshold distance traveled when an object was detected in the endpoint area immediately before and an object is currently detected in the starting area. G2S stands for Goal to Start. Specifically, N1 (G2S) is applied when an item moves to the endpoint area and then moves back to the starting area due to not being scanned, etc.

[0094] Furthermore, N1 corresponds to the threshold for the distance traveled in cases other than when an object was detected in the endpoint area immediately before and an object is currently detected in the starting area. Cases other than when an object was detected in the endpoint area immediately before and an object is currently detected in the starting area include, for example, when an object was detected in the starting area immediately before and an object is currently detected in the endpoint area. Specifically, N1 is applied when an item is scanned and then moved to the endpoint area (i.e., during a normal scan).

[0095] When there is a movement of goods from the endpoint area to the starting area, there is a high possibility that the scanned goods will change to a different goods than the goods that were detected. Therefore, N1 (G2S) > N1, and when moving from the endpoint area to the starting area, it is acceptable to make it less likely to transition to a pending state compared to other movements.

[0096] Under condition 2 of the pending state, newly detected chains become existing, and the pending chains remain.

[0097] Retention Condition 3: Product detection after a certain period of time has elapsed. The threshold for the certain period of time in Retention Condition 3 is, for example, N3 seconds (where N3 is a real number greater than or equal to 0). Note that N3 may be the same value as N2 above. If Retention Condition 3 is met, the chain of newly detected products will be added to the existing state, and the chain in the reserved state will remain. However, if the recovery conditions described later are met, it will be removed.

[0098] A chain in a pending state will be restored to an existing state if at least one or both of the following two restoration conditions are met. If an existing chain already exists, the restored chain will remain, while the existing chain may be discarded. • Restoration Condition 1: A product is detected within a certain time period starting from the time the product was last detected. • Restoration Condition 2: A product is detected within a certain distance starting from the location where the product was last detected.

[0099] Furthermore, the threshold indicating a certain time in revival condition 1 is described as R1 seconds (where R1 is a real number greater than or equal to 0), and the threshold indicating a certain distance in revival condition 2 is described as R2 (where R2 is a real number greater than or equal to 0).

[0100] Furthermore, a chain in a pending state will be deemed unrecoverable if at least one of the following unrecoverable conditions is met. In this case, the pending chain will be discarded. • Unrecoverable condition 1: No products are detected for a certain period of time or longer, starting from the time the last product was detected. • Unrecoverable condition 2: A product is detected at a location more than a certain distance away from the location where the last product was detected. • Unrecoverable condition 3: A chain pending condition occurs in another chain.

[0101] Furthermore, when discarding a chain, the information processing device 10 performs a product movement determination and a shoplifting determination, which will be described later.

[0102] (Product Movement Determination) Product movement determination refers to determining whether or not a product has moved from the starting area to the ending area. Note that product movement corresponds to a customer holding the product moving it.

[0103] A product movement is determined to have occurred if all of the following conditions are met. In other words, if all of the following conditions are met, it is determined that the product has moved from the starting area to the ending area. Condition 1: The lifespan of the chain is longer than a certain period of time. Condition 2: The final management position of the chain is the ending area. Condition 3: The product has been detected at least once in both the starting area and the ending area.

[0104] In judgment condition 1, the chain's lifespan is the period obtained by subtracting the chain's start time from the most recent time the product was detected (the chain's end time). The threshold indicating a certain period in judgment condition 1 is stated as N3 seconds (where N3 is a real number greater than or equal to 0). For example, N3 may be the lower limit, upper limit, or average value of the time required for the product to move from the starting area to the ending area.

[0105] (Shoplifting Detection) If a product movement detection is successful, the shoplifting detection will continue. In the following, a chain in which a product movement detection is successful will be referred to as a "movement successful chain."

[0106] For example, if a scan sound is recorded between the following two time points #1 and #2, it will be determined to be a normal product movement. If a scan sound is not recorded between the following two time points, it will be determined to be a possible shoplifting incident.

[0107] • Time #1: The time obtained by subtracting the frame interval margin from the chain start time of the successful movement chain. • Time #2: The latest detection time of the successful movement chain.

[0108] Time #1 corresponds to the time when the product that is the subject of the completed movement chain begins moving from the starting area. However, since camera images are taken at predetermined frame intervals, the product may start moving at a time when it is not being photographed by the camera. In this case, the start time of the chain will be later than the time when the product actually started moving. If the start time of the chain is later than the time when the product actually started moving, there is a risk that the product will be scanned before the start time of the chain, even though the scan was performed after the time when the product actually started moving. Therefore, in this embodiment, time #1 is set to the start time of the chain with a margin added (or removed). That is, by subtracting a margin from the start time of the chain, the system adjusts the time so that time #1 is definitely before the time when the product actually started moving. Time #2 corresponds to the time when the product that is the subject of the completed movement chain completes its movement to the ending area.

[0109] The shoplifting detection described herein is based on the scanning sound, but this disclosure is not limited to this. Shoplifting detection may also be performed based on the results of scene detection instead of the scanning sound. For example, shoplifting detection may be performed based on the register state (register processing state) between the two time points #1 and #2 described above. The register state will be described below.

[0110] (Cash Register Status) The shoplifting detection system (e.g., information processing device 10) manages the following boolean values ​​as an example of the cash register status. In other words, the shoplifting detection system (e.g., information processing device 10) manages whether or not it is in the checkout state and whether or not it is in the no-barcode handling state using boolean values. Here, the checkout state refers to the state from the start of checkout until the end of checkout, and may also be called the checkout in progress or the checkout state. The no-barcode handling state indicates the state in which products without barcodes are being handled. In this embodiment, this corresponds to the state in which a screen for manually selecting products without barcodes is detected. ・Checkout state ・No-barcode handling state

[0111] The truth value for the accounting status is true if at least one of the following conditions is met: • When a transition to the accounting start screen is detected by scene detection. • When a normal product movement is determined.

[0112] The truth value of the accounting status is set to false if the scene detection detects a transition to the accounting completion screen.

[0113] The truth value for the "no barcode" status is set to true if the scene detection detects a transition to the "no barcode" product screen.

[0114] If the handling status without a barcode is determined to be normal, the truth value will be set to false.

[0115] (Conditions for triggering a shoplifting alert) When a shoplifting incident is detected, the triggering of a shoplifting alert is determined based on the register status and / or the chain status. For example, in the following three cases, a shoplifting alert will not be triggered. In addition, in cases other than the following three, it may be determined that a shoplifting incident has occurred and a shoplifting alert may be triggered. If the chain's product movement detection is successful and no barcode scan is detected within a predetermined time, it is determined that a shoplifting incident has occurred.

[0116] If the "checkout status" in the register is false, the system will determine that the goods were moved while the transaction was not in progress and will not trigger a shoplifting alert.

[0117] If the register status is "No Barcode Handling Status," the system determines that the item was moved while the customer was manually entering the information for the item without a barcode, and therefore does not trigger a shoplifting alert.

[0118] If the chain's lifespan is longer than a certain period, it is determined that the movement of an object not subject to shoplifting detection has occurred, and a shoplifting alarm is not triggered. The threshold period used to compare with the chain's lifespan is specified as s1 seconds (where s1 is a non-negative real number).

[0119] Furthermore, as a backup for the microphone, if all of the following conditions are met, it may be considered that a scan sound has been detected. Also, as an alternative to the microphone scan sound, if the following conditions are met, it may be determined that a barcode scan has been detected: - The handling status without a barcode is false. - Scene detection detects a transition to the product selection screen.

[0120] (Determination of Shoplifting Possibility) In the shoplifting detection algorithm described above, there is a possibility that actions and / or behaviors of a customer that are not shoplifting may be judged as shoplifting. Therefore, in this embodiment, when an action is judged as shoplifting, a process is performed to determine the certainty of that judgment. Hereinafter, the certainty of a judgment when an action is judged as shoplifting will be referred to as "shoplifting possibility," and the process of determining shoplifting possibility will be referred to as the shoplifting possibility determination process.

[0121] In this embodiment, if the certainty of a determination that an act of shoplifting has occurred is above a predetermined level, it is referred to as "high probability of shoplifting." If the certainty of a determination that an act of shoplifting has occurred is below a predetermined level, it is referred to as "low probability of shoplifting." In this embodiment, two types of shoplifting probability are determined: "high probability of shoplifting" or "low probability of shoplifting," but this disclosure is not limited to this. For example, three types of shoplifting probability may be determined: "high probability of shoplifting," "moderate probability of shoplifting," or "low probability of shoplifting," or four or more types of shoplifting probability may be determined.

[0122] Furthermore, a low probability of shoplifting can correspond to a high probability that the determination that an act is shoplifting is incorrect. A high probability of shoplifting can correspond to a low probability that the determination that an act is shoplifting is incorrect.

[0123] The following describes the method for determining the possibility of shoplifting in the information processing device 10. The information processing device 10 determines the possibility of shoplifting using at least one of the following three determination methods.

[0124] Determination Method 1: The possibility of shoplifting is determined based on the starting position of the object's movement around the self-checkout counter. In the following, the movement of an object around the self-checkout counter may be referred to as "checkout movement." However, the scope of checkout movement is not particularly limited. For example, the movement of an object within the range included in the camera image may be referred to as checkout movement. For example, if the movement of an item starts from a designated area, it may be determined that the possibility of shoplifting is low. Conversely, if the movement of an item starts from an area other than the designated area, it may be determined that the possibility of shoplifting is high. More specifically, if a customer takes a wallet out of their bag, that wallet may be detected as an item. To prevent false shoplifting in such cases, for example, if the movement of an object (the wallet in the above example) starts from an area where a bag is expected to be placed (an example of a designated area), it is advisable to determine that the possibility of shoplifting is low. The designated area is set outside the range of the checkout counter. For example, the area includes at least one of the following: an area where a cashier (e.g., a customer) holding personal belongings may be visible; an area where personal belongings are placed; an area where personal belongings that are highly likely to be mistaken for merchandise are most likely to appear first; and an area where customers are likely to stand during checkout, such as behind the cash register. Personal belongings that are highly likely to be mistaken for merchandise include, for example, a customer's wallet or smartphone.

[0125] Determination method 1 can reduce the misjudgment that the movement of objects other than merchandise constitutes shoplifting.

[0126] Determination Method 2: The likelihood of shoplifting is determined based on the time the product spends moving around the area surrounding the register. For example, if the time the product spends moving around the register is longer than a predetermined time, it is determined that the likelihood of shoplifting is low. However, if the time the product spends moving around the register is shorter than the predetermined time, it may be determined that the likelihood of shoplifting is high. When shoplifting occurs, the scanning process is not performed, so the time the product spends moving around the register is relatively short. Also, when shoplifting occurs, it is highly likely that the product will be quickly moved into a bag or similar place to avoid being noticed by store staff. Therefore, the likelihood of shoplifting can be determined based on the length of time the product spends moving around the register. In addition, if it is detected that the product does not pass through a specific area, the likelihood of shoplifting may also be determined based on the time the product spends moving around the register. Here, the specific area is set, for example, to an area within a predetermined distance from where the scanner is installed. For example, the location of the product is detected from the camera image, and it is determined whether or not it is included in the specific area.

[0127] Method 2 of the determination process can reduce the misjudgment that a customer's act of bagging scanned items constitutes shoplifting.

[0128] Determination Method 3: Shoplifting possibility is determined based on the number of items moving around the register. For example, shoplifting possibility is determined based on whether or not multiple items are moving around the register at a specific time (or within a specific time period). For example, a score is calculated for each detected item. If multiple items with a score of a predetermined value or higher are detected at a specific time (or within a specific time period), it is determined that the possibility of shoplifting is low. If only one item with a score of a predetermined value or higher is detected, it is determined that the possibility of shoplifting is high. The score calculated here indicates the certainty of the detection result that the scanned item moved around the register. The higher the score, the higher the probability that the scanned item moved around the register. For example, as mentioned above, if multiple items with a score of a predetermined value or higher are detected at a specific time (or within a specific time period), it corresponds to a high probability that the scanned item moved around the register at a specific time (or within a specific time period).

[0129] Method 3 of the determination process can reduce the likelihood of misjudging a situation as shoplifting, for example, when multiple customers, such as a family, are paying at the same register and each customer moves items at the same time (or within the same time frame).

[0130] The information processing device 10 performs a shoplifting possibility determination as described above, and if it determines that there is a high probability of shoplifting, it implements shoplifting prevention measures, and if it determines that there is a low probability of shoplifting, it does not implement shoplifting prevention measures. Alternatively, the information processing device 10 performs a shoplifting possibility determination and implements different shoplifting prevention measures depending on whether it determines that there is a high probability of shoplifting or a low probability of shoplifting. The information processing device 10 may perform the shoplifting possibility determination using one of the three determination methods described above. The selection method is not particularly limited, but for example, a store employee may make the selection. The information processing device 10 may also perform the shoplifting possibility determination using each of the three determination methods described above and determine the shoplifting possibility from the results of each determination. For example, if all three determination results obtained using the three determination methods indicate a low probability of shoplifting, it may be determined that there is a low probability of shoplifting, and if at least one of the three determination results indicates a high probability of shoplifting, it may be determined that there is a high probability of shoplifting.

[0131] Figure 6 is a flowchart showing an example of the processing flow when performing a shoplifting possibility determination. The process shown in Figure 6 is initiated, for example, when a customer who intends to pay for items to be purchased approaches the register and the display on the display 4 in the camera image is detected to have changed, or when an item is detected in the camera image. Whether or not a customer has approached the register may be determined, for example, based on whether a human body is visible in the camera image, or based on the detection result of a human presence sensor installed at the register. In addition, in the process shown in Figure 6, the information processing device 10 acquires camera images at predetermined intervals, performs object detection processing, and determines the processing status of the register. The process shown in Figure 6 is executed based on the results of the object detection processing and the results of the scene detection.

[0132] The information processing device 10 determines whether or not all of the items to be purchased have been scanned (S101). For example, if the result of scene detection indicates that the display 4 is showing the checkout completion screen, the information processing device 10 determines that all of the items to be purchased have been scanned. Conversely, if the result of scene detection indicates that the display 4 is showing a screen different from the checkout completion screen, the information processing device 10 determines that all of the items to be purchased have not been scanned.

[0133] If all scans are complete (YES in S101), meaning there are no planned purchase items that have not yet been scanned, the flow ends.

[0134] If all scans are not complete (NO in S101), that is, if there are products to be purchased that have not been scanned, the information processing device 10 determines whether or not the scan for the product to be registered (hereinafter referred to as product #k) has been performed (S102). For example, if the result of scene detection indicates that the display 4 is showing the product selection screen, the information processing device 10 determines that the scan for product #k has been performed.

[0135] If a scan is performed (YES in S102), the information processing device 10 determines whether or not it has detected that product #k has moved to the endpoint area (S103).

[0136] If it is not detected that product #k has moved to the endpoint area (NO in S103), the flow returns to S103, and object detection processing is performed until product #k moves to the endpoint area.

[0137] If it is detected that product #k has moved to the endpoint area (YES in S103), the flow returns to S101.

[0138] If a scan has not been performed (NO in S102), the information processing device 10 determines whether or not it has detected that product #k has moved to the endpoint area (S104).

[0139] If it is not detected that product #k has moved to the final area (NO in S104), the flow returns to S101.

[0140] If the system detects that product #k has moved to the end area (YES in S104), the information processing device 10 executes a shoplifting possibility determination process for product #k (S105).

[0141] The information processing device 10 determines whether there is a high probability of shoplifting for product #k (S106).

[0142] If the likelihood of shoplifting for product #k is not high (NO in S106), the flow returns to S101. In this case, the information processing device 10 does not implement any measures to prevent shoplifting.

[0143] If there is a high probability of shoplifting for product #k (YES in S106), the information processing device 10 implements measures to prevent shoplifting (S107). Then the flow returns to S101.

[0144] As described above, by performing a shoplifting probability determination process to assess the certainty of a shoplifting determination when it is determined that an act has occurred, the occurrence of false detections of shoplifting can be suppressed. Furthermore, in cases where the probability of an act being shoplifting is low, that is, when the certainty of a shoplifting determination is lower than a predetermined level, measures to prevent shoplifting are not taken, thus avoiding the execution of unnecessary processes. For example, it is possible to avoid notifying customers and / or store employees as a measure to prevent shoplifting, thereby reducing unnecessary notifications.

[0145] (Method of notifying shoplifting) In the shoplifting detection algorithm described above, if shoplifting is detected, the detection result is notified to the customer and / or the store clerk. This notification method is not particularly limited, but for example, the customer is notified of the detection result by the information processing device 10 displaying text information or the like on the display 40. Alternatively, the store clerk is notified of the detection result by the information processing device 10 transmitting information about the detection result to a terminal operated by the store clerk. The notification here includes at least one of the following: notification of information using the display of text information or the like on a display, notification of information using the output of sound from a speaker, and notification of information using the lighting, flashing, or color change of a light or lamp such as a warning light.

[0146] If shoplifting is detected, both the customer and the store employee are notified, but in this embodiment, different notification methods are applied to the customer and the store employee, respectively. For example, when the information processing device 10 detects shoplifting, it controls the notification method for notifying the customer of the detection result (e.g., a first notification method) and the notification method for notifying the store staff of the detection result (a second notification method) to be different. Notification of the detection result to the customer is performed, for example, by controlling the display 40. Notification of the detection result to the store staff is performed, for example, by controlling a terminal operated by the employee. The control here includes sending the information to be notified, instructing the notification method, stopping the notification, etc.

[0147] The following describes an example of a notification method executed by the information processing device 10.

[0148] Notification Method 0-1: For example, if shoplifting is detected, the information processing device 10 notifies either the customer or the store clerk of the detection result, but not the other. If shoplifting is detected, the customer is notified of the detection result, but the store clerk is not. In this way, shoplifting can be deterred without bothering the store clerk who is managing multiple self-checkout machines. Alternatively, if shoplifting is detected, the customer is notified of the detection result, but the store clerk is notified. In this way, store clerks can be vigilant against shoplifting while preventing the customer's feelings from being hurt when they made an error in operation or otherwise did not intend to shoplift.

[0149] Notification method 0-2: For example, if shoplifting is detected, the method of notifying the customer and the store employee of the detection result will be different.

[0150] Notification Method 1: If shoplifting is detected and the likelihood of shoplifting is deemed low, the detection result will be notified to the store staff, but not to the customer. This prevents notifying customers who have not actually shoplifted, while still informing store staff that there is a possibility of shoplifting occurring.

[0151] Notification Method 2: In Notification Method 1, the method of notifying store employees may be changed depending on whether the likelihood of shoplifting is low or high. For example, if the likelihood of shoplifting is low, only the history information of the shoplifting activity may be stored on the terminal operated by the store employee, and no notification may be sent. This prevents store employees from being bothered by notifications even when the likelihood of shoplifting is low.

[0152] Notification Method 3: If shoplifting is detected and it is determined that the likelihood of shoplifting is low, the information processing device 10 notifies both the store clerk and the customer. In this way, both the store clerk and the customer can recognize that an act with a low likelihood of shoplifting has occurred, making it easier for the store clerk to remind the customer to be careful when operating the self-checkout machine. Different notification methods and / or display methods may be applied between the notification to the store clerk and the notification to the customer. In this case, for example, the notification to the customer may include a notice that there is a risk of error, and avoid notifications that would intimidate the customer, such as a notice that shoplifting may have occurred.

[0153] Notification Method 3-1: As an example of Notification Method 3, the information processing device 10 changes the display format based on the number of shoplifting incidents, but only in the case of notifications to customers, compared to notifications to store staff. For example, the color and size of the display frame, the type of font, the font size, the font color, etc., are changed according to the number of shoplifting incidents. This allows for a strong warning to be issued to customers who continue to engage in suspicious shoplifting activities even after being notified. The number of shoplifting incidents here corresponds to the number of items that were not scanned, or the number of times the customer did not scan items. In this case, the notification to store staff can be a notification that there is a possibility of shoplifting, regardless of the number of shoplifting incidents. This is because store staff manage the status of multiple self-checkout registers in the store, and changing the display on the display that store staff check according to the number of shoplifting incidents may make it difficult for store staff to grasp the overall status of the store. However, if it is necessary to notify store staff of the priority of customer service, etc., the notification format may be changed according to the number of shoplifting incidents. For example, if prioritizing the response of customers who frequently shoplift over those who shoplift infrequently, the manner of notification may be changed according to the number of shoplifting incidents.

[0154] Notification Method 3-2: As an example of Notification Method 3, for instance, in the case of notifications to customers rather than to store staff, an audio notification is given only to customers when the number of shoplifting incidents reaches a predetermined number. In Notification Method 3-2, if the number of shoplifting incidents has not reached a predetermined number, an audio notification is not required. Since audio notifications can be heard by other customers in the vicinity, there is a risk that customers other than the customer who committed the shoplifting incident may also be intimidated. Therefore, with this method, by giving an audio notification only when there is a malicious customer who is suspected of repeatedly shoplifting, it is possible to prevent other customers from being excessively intimidated.

[0155] Notification method 3-3: In notification method 3-2, at least one of the following is changed depending on the number of times shoplifting has occurred: the content of the voice message and the volume of the sound.

[0156] Notification Method 3-4: Of the two methods of notification, one for store staff and one for customers, only the notification to customers involves illuminating a light or similar device installed at the register when the number of shoplifting incidents reaches a predetermined number. In notification method 3-4, if the number of shoplifting incidents has not reached the predetermined number, it is not necessary to illuminate the light or similar device installed at the register. Since notifications using lights or similar devices are also noticeable to other customers in the vicinity, there is a risk that customers other than the customer who committed the shoplifting incident may also be intimidated. Therefore, with this method, by using lights or similar devices to notify only when there is a malicious customer who is suspected of repeatedly shoplifting, it is possible to prevent other customers from being excessively intimidated.

[0157] Notification method 3-5: In notification method 3-4, at least one of the following is changed depending on the number of times shoplifting has occurred: the color of the light to be illuminated, the brightness of the light, whether or not it flashes, the speed of flashing, etc.

[0158] Notification Method 4: If shoplifting is detected and it is determined that there is a high probability of shoplifting, the information processing device 10 notifies both the store clerk and the customer. Different notification methods and / or display methods may be applied to the notification to the store clerk and the notification to the customer. Notification methods 3-1 to 3-5 may be used to notify the customer.

[0159] Furthermore, in notification method 4, if it is determined that there is a high probability of shoplifting, a different notification method and / or display method may be applied compared to when it is determined that there is a low probability of shoplifting. For example, if it is determined that there is a high probability of shoplifting, the notification to the customer may be louder than when it is determined that there is a low probability of shoplifting, or the content of the notification may be stricter than when it is determined that there is a low probability of shoplifting. In addition, the method of notifying store employees may be changed depending on whether there is a high or low probability of shoplifting. For example, only store employees may be notified whether there is a high or low probability of shoplifting.

[0160] Furthermore, in notification method 4, if the likelihood of shoplifting is determined to be low, different notifications may be sent to the store clerk and the customer, and if the likelihood of shoplifting is determined to be high, the same notification may be sent to both the store clerk and the customer. For example, if the likelihood of shoplifting is determined to be low, there is a possibility that the shoplifting activity has been falsely detected, so in order to avoid harming the customer's perception, direct expressions such as "shoplifting" should not be used in the notification to the customer, while the notification to the store clerk should be sent directly to inform them that there is a possibility of shoplifting in order to encourage caution. On the other hand, if the likelihood of shoplifting is determined to be high, it is effective to directly notify the customer that there is a possibility of shoplifting and issue a warning, so the same notification that there is a possibility of shoplifting may be sent to both the customer and the store clerk.

[0161] (Example of notification method) Figure 7 is a diagram showing an example of a notification method. Figure 7 shows an example of a message displayed on the display 40 as an example of notifying a customer of the results of shoplifting detection. Figure 7 shows an example of the method 3-1 described above, in which the display method is changed depending on the number of times shoplifting has occurred (for example, the number of items that were not scanned). As shown in Figure 7, the display method of the message displayed on the display 40 is different for the case of one shoplifting incident, two incidents, and three incidents.

[0162] For example, if shoplifting has only occurred once, the message will be displayed in a relatively inconspicuous manner. In this case, audio notification is not required.

[0163] If shoplifting occurs twice, the message will be displayed in a way that is relatively more conspicuous compared to when it occurs once. In this case, the text and frame of the message may be enlarged, and the color of the message may be changed. In addition, in this case, an audio notification may be provided.

[0164] If shoplifting has occurred three times, the message will be displayed in a more prominent manner compared to the cases of one or two shoplifting incidents. The display position may also be changed in this case. Audio notification may also be provided. Furthermore, the text of the message itself may be changed compared to the case of two shoplifting incidents. Audio notification may also be provided in this case.

[0165] By applying the notification methods described above, appropriate notifications can be made to both customers and store staff. For example, customers can be clearly notified that shoplifting has occurred, and appropriate warnings can be given to them. For store staff, the workload of managing multiple self-checkout registers can be reduced by limiting the information to be notified.

[0166] (Measures to prevent false detection in scene detection) As described above, the information processing device 10 uses a learning model to identify the display on the self-checkout monitor (for example, display 4 in Figure 1) and detects displays that are labeled with labels such as "checkout start screen," "checkout end screen," "no barcode product screen," and "product selection screen."

[0167] However, there is a possibility of false detection of scenes. For example, if the "checkout completion screen" is detected during the checkout process, shoplifting detection will not be possible in the subsequent checkout process. Also, if the self-checkout monitor is obscured by the customer's body, the monitor display cannot be detected, and there is a risk that the change in scene will not be detected. Therefore, this embodiment describes a method to suppress the decrease in shoplifting detection accuracy due to false detection of scenes.

[0168] Method 0: For example, the information processing device 10 determines to continue the shoplifting detection process if the following two conditions are met: • Condition 1: The movement of the product to the register is detected. • Condition 2: The barcode has been scanned.

[0169] Whether or not judgment condition 1 is met is determined based on the results of product detection. Furthermore, whether or not judgment condition 2 is met, that is, whether or not the barcode has been scanned, is determined by detecting the scan sound or the register screen at the time of scanning (for example, the product selection screen).

[0170] In this embodiment, the information processing device 10 is assumed to be unable to directly acquire information indicating the current state of the self-checkout machine. Therefore, the information processing device 10 estimates the state of the self-checkout machine from the monitor display and scanning sounds. However, if it is possible to acquire information indicating the current state from the self-checkout machine, the information processing device 10 may recognize the state of the self-checkout machine based on that information. The information indicating the current state of the self-checkout machine (for example, the scene described above) may be acquired directly from, for example, the information processing device 1 in Figure 1.

[0171] Method 1: In the judgment condition 2, if a screen for manually entering the quantity of a product is detected, it may be determined that the judgment condition 2 is satisfied. This is because when dealing with products without barcodes, manual entry is performed instead of barcode scanning.

[0172] Method 2: After the checkout completion screen is detected, if the two judgment conditions described above are met within a predetermined time frame, it is determined that the shoplifting detection process will continue. If the two judgment conditions described above are met within a predetermined time frame after the checkout completion screen is detected, the customer is still in the process of registering items, so the detected checkout completion screen is likely to be a false positive. According to Method 2, it is possible to avoid stopping the shoplifting detection process due to this false positive and to continue the shoplifting detection process. In addition, in Method 2, if the two judgment conditions described above are met after the checkout completion screen is detected and after a predetermined time has elapsed, it may be determined not to continue the shoplifting detection process.

[0173] The above explanation described a case where the "accounting completion screen" is detected during the accounting process, but this disclosure is not limited to this.

[0174] Method 3: For example, if the checkout start screen is not detected and the two conditions above are met, the shoplifting detection process may be started. If the checkout start screen is not detected and the two conditions above are met, it corresponds to a case where the customer has started registering items, but the checkout start screen may not have been detected. In other words, there is a possibility that the detection of the checkout start screen has failed. Method 3 can reduce the error of the shoplifting detection process not starting even though the customer has started the checkout process (for example, the process of registering items).

[0175] Method 4: For example, if, after the checkout start screen is detected, the shoplifting detection process may be stopped if at least one of the above two judgment conditions remains unmet for a predetermined period of time. Alternatively, if, after the checkout end screen is detected, the shoplifting detection process may be stopped if at least one of the above two judgment conditions remains unmet for a predetermined period of time. If the customer is continuing to register items, both of the above two judgment conditions will be met. Therefore, if at least one of the conditions remains unmet for a predetermined period of time, it is highly likely that the item registration has already been completed. As a result, the possibility of shoplifting disguised as item registration is low, and the possibility of missing a shoplifting incident even if the shoplifting detection process is stopped is low.

[0176] Method 5: If the checkout start screen is detected, and a predetermined period of time has passed during which at least one of the above two judgment conditions is not met, then the shoplifting detection process may be started. If the checkout start screen is detected, and a predetermined period of time has passed during which at least one of the above two judgment conditions is not met, then this corresponds, for example, to a situation where the start of the next customer's checkout is detected before the completion of the previous customer's checkout has been detected. Therefore, in Method 5, the shoplifting detection process is started to detect the next customer's shoplifting. Method 5 allows for the appropriate detection of the next customer's shoplifting.

[0177] Methods 0 and 1-3 described above are also effective when the accounting start screen is not detected, and / or when the accounting end screen is mistakenly detected during accounting.

[0178] Furthermore, methods 4 and 5 described above are effective when the accounting completion screen is not detected, and / or when the accounting start screen is mistakenly detected after accounting is completed.

[0179] The above-mentioned countermeasures against false detections in scene detection prevent shoplifting detection from being failed due to false detections of scenes, thereby suppressing a decrease in the accuracy of shoplifting detection.

[0180] (Example of a processing flow for preventing false positives in scene detection) Figure 8 is a flowchart showing an example of a processing flow for preventing false positives in scene detection. Note that in Figure 8, processes similar to those in Figure 6 are given the same reference numbers and explanations may be omitted.

[0181] In Figure 8, if all scans are not completed (NO in S101), that is, if the checkout completion screen is not detected, the information processing device 10 determines whether or not it has detected the movement of a certain product (S201).

[0182] If the movement of a certain product is detected (YES in S201), the flow proceeds to S102.

[0183] If the movement of a certain product is not detected (NO in S201), the information processing device 10 determines whether a predetermined period of time or longer has passed during which at least one of the two judgment conditions described above—namely, the judgment condition that the movement of the product to the register is detected, and the judgment condition that the barcode has been scanned—has not been met (S202).

[0184] If the period during which at least one of the two judgment conditions is not met does not continue for a predetermined amount of time (NO in S202), that is, if product movement or barcode scanning is detected, the flow proceeds to S102.

[0185] If at least one of the two judgment conditions remains unmet for a predetermined period of time or longer (YES in S202), the shoplifting detection process shown in Figure 8 is terminated. This case corresponds to situations where the payment has been completed but the payment completion screen has not been detected, or where the payment start screen has not been detected after the payment has been completed.

[0186] In Figure 8, when all scans are completed (NO in S101), that is, when the accounting completion screen is detected, the information processing device 10 determines whether a predetermined period of time or longer has passed during which at least one of the two judgment conditions described above has not been met (S203).

[0187] If the period during which at least one of the two judgment conditions is not met does not continue for a predetermined amount of time (NO in S203), that is, if product movement or barcode scanning is detected, the flow proceeds to S101. This case corresponds to a case where the accounting completion screen is incorrectly detected in S101 despite accounting processing being in progress, or where the accounting start screen is not detected.

[0188] If the period during which the two judgment conditions are not met continues for a predetermined time or longer (YES in S203), the information processing device 10 terminates the shoplifting detection process shown in Figure 8.

[0189] As explained above, as a countermeasure against false detection of scenes, determining the register status based on at least one of the time periods when products are not being moved and the time periods when barcodes are not being scanned can prevent shoplifting detection from being incorrectly stopped and suppress a decrease in the accuracy of shoplifting detection.

[0190] (Method for disabling movement detection) In the shoplifting detection algorithm described above, there is a possibility that customer actions and / or movements that are not shoplifting may be judged as shoplifting. Therefore, in this embodiment, we will explain how to reduce false judgments of shoplifting by disabling the detection of movement when certain conditions are met in object detection, or more specifically, the detection of object movement. Note that the detection of object movement refers to the detection that the detected object is moving around the cash register. Note that the object to be detected here is not limited to merchandise, but may be an object other than merchandise.

[0191] The following describes the method for disabling movement detection, which is executed when the information processing device 10 detects the movement of a product during the product detection process. The information processing device 10 determines whether or not to disable the detection of object movement based on the method shown below. The conditions for determining whether or not to disable are referred to as the "disable conditions." If the "disable conditions" are met, the information processing device 10 disables the detection of object movement. Disabling the detection of object movement means discarding the object movement detection result and treating the object as if it had not moved, even though it actually did. When the detection of object movement is disabled, the information processing device 10 returns the state of the managed object to a state where no object movement has been detected. That is, the information processing device 10 continues processing such as the product detection process as if the movement of the object has not yet started. By performing such processing, for example, if a customer returns an object they were holding and picks up another object, the possibility that the information processing device 10 might mistakenly detect these different objects held by the customer as the same single object can be reduced.

[0192] Disabling Method 1: If it is detected that the movement of an object started from a specific area, the movement detection is considered disabled. The condition for disabling Method 1 is that it is detected that the movement of an object started from a specific area. The specific area here may be, for example, the area behind the cash register where customers normally stand (for example, the personal belongings area in Figure 2B). In this case, it corresponds to, for example, the detection of movement of a customer's personal belongings.

[0193] Disabling Method 1-1: In Disabling Method 1, the specific area may be changed dynamically. For example, the specific area may be changed depending on whether a person or something other than a person is present in that area. Here, "something other than a person" specifically refers to things like carts used to move goods in a store. For example, even if the goods are being moved from the same location, if a cart is placed in that location, the goods are more likely to be merchandise, and if there is no cart, they are more likely to be personal belongings. Therefore, if a cart is present, this location should be excluded from the specific area, and if a person is present, this location should be included in the specific area. Note that whether a person or something other than a person is present can be identified using existing image recognition technology.

[0194] Disabling Method 1-2: In disabling method 1-1, the timing of setting the specific area may be changed dynamically. For example, the specific area may be set when the start of payment is detected. Note that payment is determined to have started when the camera image detects that the register screen has become the payment start screen. Alternatively, the specific area may be set after the payment start screen has been detected and when the payment start screen is no longer detected. Alternatively, the specific area may be set when the first barcode scan is detected. Note that barcode scans are detected by the scan sound and / or changes in the register screen. If the specific area is set at a time when it is unlikely that the customer will take out personal belongings, there is a risk that the specific area could be misused to commit shoplifting. Therefore, by setting the specific area at the start of payment or when the payment process is more advanced, the possibility of the specific area being misused can be reduced.

[0195] Invalidation Method 2: When movement of an object is detected exceeding a predetermined distance within a unit time, it is determined that an object other than the target object (the object being tracked) has been detected. In this case, the detection of movement exceeding the predetermined distance is deemed invalid, and the detection of object movement is terminated. The invalidation condition for Invalidation Method 2 is that movement exceeding a predetermined distance is detected within a unit time. When such movement is detected, for example, there is a high possibility that multiple different objects located far apart from each other are being mistakenly identified as the same object. Therefore, the shoplifting determination result and the detection of object movement will be deemed invalid if a shoplifting act is determined based on a erroneously detected movement.

[0196] In invalidation method 2, the predetermined distance when the object is moving in the forward direction and the predetermined distance when the object is moving in the reverse direction may be different from each other. Here, the forward direction corresponds to the direction in which an item normally moves when a customer scans an item in the accounting process, as shown in Figure 3. The forward direction corresponds to the direction from the item stand A1a before scanning to the item stand A1b after scanning. The reverse direction is the opposite direction of the forward direction, as shown in Figure 3. The reverse direction is the direction from the item stand A1b after scanning to the item stand A1a before scanning.

[0197] For example, the predetermined distance for movement in the reverse direction is set to a smaller value than the predetermined distance for movement in the forward direction. In this case, even if no false detection of an object occurs, movement detection in the reverse direction is more likely to be deemed invalid.

[0198] Movement in the reverse direction may occur, for example, when returning an item to the item stand A1a before scanning and attempting to scan another item first. Therefore, to ensure accurate detection of other item movements, it is beneficial to disable movement in the reverse direction even if no object misdetection occurs. In this example, the predetermined distance for reverse movement is set to a smaller value than the predetermined distance for forward movement, making it easier to determine that another item has been detected.

[0199] If it is difficult to determine whether the movement is forward or reverse, it may be treated as a direction that is neither forward nor reverse. This direction, which is neither forward nor reverse, will be described as a third direction. The third direction could be, for example, a direction that is approximately perpendicular to both the forward and reverse directions. In the case of forward movement, the customer moves the product along a trajectory that makes it easy to scan, and the possibility of it being detected as a third direction is low. Therefore, the predetermined distance in the third direction may be the same as the predetermined distance in the reverse direction.

[0200] A predetermined distance may be set for the forward direction, the reverse direction, and the third direction. For example, the three predetermined distances may be set such that the predetermined distance in the forward direction is the largest and the predetermined distance in the reverse direction is the smallest.

[0201] Deactivation Method 3: In the case of object movement, if no object movement is detected within a predetermined time, the object movement detection is considered invalid. The deactivation condition for Deactivation Method 3 is that no object movement is detected within a predetermined time.

[0202] Invalidation Method 4: If the movement of an object is detected to have been interrupted and then the start of its movement is detected again, it is determined whether the movement of the same object has resumed based on the object's position at the time the movement was interrupted and its position at the time the movement resumed. For example, if the distance between the object's position at the time the movement was interrupted and its position at the time the movement resumed is less than a predetermined value, it is determined that the movement of the same object has resumed. Method 4 is invalid if the distance between the object's position at the time the movement was interrupted and its position at the time the movement resumed is greater than or equal to a predetermined value. This applies, for example, when an object is temporarily placed down and another object is picked up.

[0203] In invalidation method 4, if the distance between the position of the object at the time the movement was interrupted and the position of the object at the time the movement resumed is less than a predetermined value, it is determined that the object before the movement was interrupted and the object at the time the movement resumed are the same object, and that the movement of that same object has resumed.

[0204] Furthermore, in invalidation method 4, the time of the object's movement may be measured from the moment the movement restarts. In other words, the movement time measured before the movement restarts and before the movement is interrupted does not need to be included.

[0205] In invalidation method 4, if the distance between the object's position at the time the movement was interrupted and the object's position at the time the movement resumed is greater than or equal to a predetermined value, there is a possibility that the object before the movement was interrupted and the object at the time the movement resumed are not the same object. Therefore, in invalidation method 4, it is determined that the movement of the object at the time the movement resumed is the movement of a different object from the object before the movement was interrupted, and the interrupted product movement is deemed invalid and discarded. Here, invalidation is determined based on the distance between the object's position at the time the movement was interrupted and the position at which the movement resumed, but this disclosure is not limited to this. For example, if the difference between the time the movement was interrupted and the time the movement resumed is greater than or equal to a predetermined value, the interrupted product movement may be deemed invalid and discarded.

[0206] (Example of the process for disabling motion detection) Figure 9 is a flowchart showing an example of the process for disabling motion detection. Note that in Figure 9, processes similar to those in Figures 6 and 8 are given the same reference numbers and their explanations may be omitted.

[0207] In Figure 9, if the system detects that product #k has moved to the endpoint area (YES in S104), the information processing device 10 executes a process to invalidate the detection of the movement of product #k (S301). Here, the information processing device 10 determines whether or not to invalidate the movement of product #k based on whether or not the invalidation conditions are met.

[0208] The information processing device 10 determines whether or not the invalidation condition is met for product #k (S302).

[0209] If the invalidation condition is met for item #k (YES in S302), the information processing device 10 determines that the movement of item #k is invalid, and the flow returns to S101. In this case, the information processing device 10 does not implement measures to prevent shoplifting.

[0210] If the invalidation condition is not met for item #k (NO in S302), the information processing device 10 implements measures to prevent shoplifting (S107). Note that if the invalidation condition is not met for item #k, it is determined that the movement of item #k is not invalidated, that is, it is valid. In this case, since the item was not scanned and the movement of the item that moved to the endpoint area is valid, it is determined that shoplifting of that item has occurred. The flow then returns to S101.

[0211] As described above, the process of disabling object movement detection, which determines whether or not object movement detection is disabled, can suppress the occurrence of false detections of shoplifting. For example, cases in which object movement detection is disabled include cases where the detected object is not a product, cases where the detected object is not moving, and cases where the detected movement is not that of a single common object. For example, by disabling object movement detection, it is possible to avoid the false determination that the movement of the target object constitutes shoplifting.

[0212] (Example of the configuration of the information processing device 10) Figure 10 is a block diagram showing an example of the configuration of the information processing device 10. The information processing device 10 shown in Figure 10 has a receiving unit 101, a control unit 102, and a transmitting unit 103. The control unit 102 includes an object detection unit 104, a feature point detection unit 105, a sound detection unit 106, a state determination unit 107, a shoplifting determination unit 108, a display information generation unit 109, a display control unit 110, and a transmission control unit 111. The control unit 102 may be configured by a processor such as a CPU (central processing unit).

[0213] The receiving unit 101 acquires camera images from, for example, the camera 20. For example, if the camera 20 includes an object detection camera 20 and a feature point detection camera 20, the receiving unit 101 acquires an object detection camera image captured by the object detection camera 20 and a feature point detection camera image captured by the feature point detection camera 20. Note that the object detection camera image and the feature point detection camera image may be the same camera image. In other words, the camera 20 may capture an image that can detect feature points on the screen displayed on the display 4 and can also detect products being moved by the customer.

[0214] The receiving unit 101 may also acquire sound from the microphone 30.

[0215] The object detection unit 104 performs product detection processing based on the camera image (for example, the camera image for object detection). Product detection processing may also be referred to as object detection processing. The object detection unit 104 outputs object detection information regarding the detected object to the shoplifting detection unit 108. The object detection information includes information indicating whether or not the object detection unit 104 detected an object. Furthermore, if the object detection unit 104 detects the start of movement of an object, the object detection information may include information indicating the timing of the start of movement of the detected object. The object detection information may also include the direction of movement of the detected object, the destination, the position where the movement was completed, etc. The object detection unit 104 may disable the detection of product movement based on the movement detection disabling processing described above.

[0216] The feature point detection unit 105 detects feature points in the camera image. For example, the feature point detection unit 105 detects feature points in the camera image displayed on the display 4 and outputs image feature point information indicating the detected image feature points to the state determination unit 107.

[0217] The voice detection unit 106 detects sounds and other noises that occur when a product is scanned, based on the sound picked up by the microphone 30.

[0218] The state determination unit 107 performs scene detection to determine the processing status of the register corresponding to the display on the display 4. For example, the state determination unit 107 performs scene detection based on image feature point information and determines which label the display screen of the display 4 is, such as "checkout start screen," "checkout end screen," "no barcode product screen," or "product selection screen." Based on the label, the state determination unit 107 determines whether the processing status of the register is in the checkout start state, the checkout end state, the state of registering a no-barcode product, or the state in which the purchased product has been selected by barcode reading or monitor operation (i.e., the registration state). The state determination unit 107 may also determine that the processing status of the register is in the registration state if it detects an audio sound generated when a product is scanned.

[0219] The shoplifting detection unit 108 performs shoplifting detection processing. For example, the shoplifting detection unit 108 determines whether or not shoplifting has occurred based on the register status determined by the status determination unit 107 and the object detection information obtained from the object detection unit 104. In this determination, the movement of goods may be managed based on a chain. For example, the shoplifting detection unit 108 determines that shoplifting has occurred if it indicates that the movement of a product started from the starting area, the product passed through the area around the scanner 2, and moved to the ending area, and that the product is not registered. On the other hand, the shoplifting detection unit 108 determines that shoplifting has not occurred if it indicates that the movement of a product started from the starting area, the product passed through the area around the scanner 2, and moved to the ending area, and that the product has been scanned. Note that if the product is not registered, it means that the product has not been scanned or the product information has not been manually entered. Whether or not the product has been scanned or whether or not the product information has been manually entered is determined based on the register status.

[0220] The shoplifting detection unit 108 may perform the above-described shoplifting possibility determination along with the shoplifting detection process. The shoplifting detection unit 108 may also perform the scene detection false detection processing as described above. Furthermore, the shoplifting detection unit 108 may determine the method for notifying the determination result of the shoplifting detection process based on the shoplifting act notification method as described above.

[0221] The display information generation unit 109 generates display information for displaying the shoplifting detection result of the shoplifting detection process in the shoplifting detection unit 108 on the display 40, and / or display information for displaying the detection result on the store clerk's terminal. The display information may include the content of the message to be displayed, or it may include instructions on the display manner, such as the color, font, and size of the message. The display information generation unit 109 generates the display information based on the notification method determined by the shoplifting detection unit 108. For example, if the notification method is determined to notify the store clerk but not the customer of the detection result, the display information generation unit 109 generates display information to be displayed on the store clerk's terminal, but does not generate display information to be displayed on the display 40. Also, if the notification method is determined to notify both the customer and the store clerk of the detection result, but in different display manners, the display information generation unit 109 sets the display manner instructed in the display information to be displayed on the display 40 and the display manner instructed in the display information to be displayed on the store clerk's terminal to be different display manners.

[0222] The display control unit 110 controls the display on the display 40 installed in the cash register. For example, the display control unit 110 causes the display 40 to display information according to the display information or instructions generated by the display information generation unit 109.

[0223] The transmission control unit 111 generates transmission information including display information and transmits the transmission information via the transmission unit 103. The transmission information may be transmitted, for example, to a terminal owned by a store employee, or to a display device that can be viewed by a store employee, security guard, etc.

[0224] The receiving unit 101 of the information processing device 10 described above acquires an image from the camera 20, as an example. The acquired image may be an image capturing the actions of a customer in a store, from moving an item placed in a designated starting area (an example of a first area) to a scanning area (an example of a second area) to register the item with the scanner 2 (an example of a registration device), and then moving the item from the scanning area to an ending area (an example of a third area). The control unit 102 of the information processing device 10 performs a detection process to determine whether the customer has registered an item in the scanning area based on the image acquired by the receiving unit 101. If it detects that the customer has not registered an item in the scanning area, it performs control to differentiate between a first notification method for notifying the customer of the detection result and a second notification method for notifying the store staff of the detection result.

[0225] In this embodiment, an example is shown in which an information processing device 10 provided at each self-checkout counter performs object detection processing, scene detection processing, and shoplifting detection. However, this disclosure is not limited to this. At least one of the object detection processing, scene detection processing, and shoplifting detection may be performed by other devices such as a terminal owned by a store employee or a server installed at the store. For example, if the object detection processing is performed by another device, the information processing device 10 transmits camera images, etc., to the other device and obtains the results of the object detection processing from the other device.

[0226] Furthermore, in the above embodiment, the information processing device 10 detected the movement of the product using camera images. However, the movement of the product may be detected by other means. For example, the information processing device 10 may detect the movement of the product based on changes in weight detected by weight sensors provided on product stands A1a and A1b. Alternatively, for example, if the product is fitted with a wireless tag such as an RFID, the information processing device 10 may detect the movement of the product by detecting the movement of the wireless tag. In addition, the information processing device 10 may detect the movement of the product by using a combination of camera images for object detection and detection results from weight sensors or wireless tags.

[0227] Furthermore, in the above embodiment, the information processing device 10 determined the processing status of the cash register corresponding to the display on the display 4 by detecting feature points of the image. However, the information processing device 10 may also determine the processing status of the cash register using other methods, such as recognizing characters displayed on the display 4 or using pattern matching of the entire image displayed on the display 4.

[0228] Furthermore, while the above embodiments assumed a self-checkout system, this disclosure is not limited to this. For example, this disclosure may be applied to a cash register where an employee performs scanning, or a semi-self-checkout system where the employee performs some of the accounting tasks and the customer performs some of them. In this case, it may be possible to determine whether or not the employee is committing shoplifting.

[0229] The information processing device 10 in this embodiment may be implemented as a PC and a program that operates the PC as an external device. In this case, the program may be provided by an entity other than the PC.

[0230] Furthermore, if a product requiring age verification is being scanned, a display area for age verification and display areas corresponding to "Yes" and "No" age verification buttons will be displayed to indicate that there are restrictions on purchasing the product. The information processing device 10 may detect one or more feature points from either the age verification display screen or the display screen for the barcode-scanned product name as feature points of the camera image.

[0231] When registering products that do not have barcodes, the information processing device 10 may detect one or more feature points from the screen displaying a list of manually entered products, the screen where the quantity is entered, and the screen displaying the manually entered product name as feature points of the camera image.

[0232] In the above embodiment, a scan may be determined when at least one of the following is detected: the scan sound during barcode scanning or the image feature points on the display screen of the display 4 during barcode scanning. Alternatively, a scan may be determined when both the scan sound during barcode scanning and the display screen of the display 4 during barcode scanning are detected. If the time at which the scan sound is detected and the time at which the image feature points are detected are different, time adjustment may be performed.

[0233] While the above embodiment illustrates an example of determining shoplifting of goods sold in a store, this disclosure is not limited to this. For example, this disclosure may also be applied to items that are lent, promotional goods that are distributed, items that can be taken out in limited quantities, etc.

[0234] In addition, the "...part" in the above embodiment may be a "...circuitry", a "...device", a "...unit", or a "...module".

[0235] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0236] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0237] The integrated circuit implementation method is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, a Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0238] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0239] This disclosure is applicable to all types of devices, systems, and equipment with communication capabilities (collectively referred to as communication equipment). Non-exclusive examples of communication equipment include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth and telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned equipment.

[0240] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting fixtures, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0241] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0242] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0243] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0244] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0245] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above.

[0246] The disclosures in the specification and abstract included in the Japanese application 2024-190798 filed on October 30, 2024, the disclosures in the specification and abstract included in the Japanese application 2024-190801 filed on October 30, 2024, and the disclosures in the specification and abstract included in the Japanese application 2024-190803 filed on October 30, 2024 are all incorporated herein by reference.

[0247] This disclosure is useful for detecting fraudulent activity in self-checkout systems.

[0248] 1, 10 Information processing device 2 Scanner 3, 20 Camera 30 Microphone 4, 40 Display A1a, A1b Product stand 101 Receiving unit 102 Control unit 103 Transmitting unit

Claims

1. An information processing device comprising: a receiving unit that acquires images of the actions taken by a customer in a store, from moving an item placed in a first area to a second area and registering the item using a registration device, until the item is moved from the second area to a third area; and a control unit that performs a detection process based on the images to determine whether the customer has registered the first item in the second area, and if it is detected that the customer has not registered the first item in the second area, controls the first notification method for notifying the customer of the detection result and the second notification method for notifying the store staff of the detection result to be different.

2. The information processing apparatus according to claim 1, wherein the control unit detects that the customer has not registered the first item in the second area, notifies the customer of the detection result, but does not notify the staff of the detection result.

3. The information processing apparatus according to claim 1, wherein the control unit determines the likelihood of the detection result that the customer has not registered the first item in the second area in the detection process.

4. The information processing apparatus according to claim 3, wherein if the control unit determines that the likelihood of the detection result that the customer has not registered the first item in the second area is lower than a predetermined level, it does not notify the customer of the detection result, but notifies the staff of the detection result.

5. The information processing apparatus according to claim 3, wherein the control unit provides different methods for notifying the staff of the detection result when the certainty of the detection result that the customer has not registered the first item in the second area is lower than a predetermined level, and for notifying the staff of the detection result when the certainty of the detection result is not lower than the predetermined level.

6. The information processing apparatus according to claim 3, wherein the control unit, when the certainty of the detection result that the customer has not registered the first item in the second area is lower than a predetermined level, makes the notification method for notifying the customer of the detection result different from the notification method for notifying the staff of the detection result.

7. The information processing apparatus according to claim 6, wherein the control unit changes the method of notifying the customer of the detection result according to the number of first items that the customer has not registered in the second area when notifying the customer of the detection result.

8. A control method comprising: an information processing device acquiring images of the actions taken by a customer in a store, from moving an item placed in a first area to a second area and registering the item using a registration device, to moving the item from the second area to a third area; performing a detection process based on the images to determine whether the customer has registered the first item in the second area; and, if it is detected that the customer has not registered the first item in the second area, controlling the device to use different methods for notifying the customer of the detection result (first notification method) and notifying the store staff of the detection result.

Images