Shelf state detection device and shelf state detection method

Abstract

The present invention detects a stockout on a shelf on which a plurality of products are arranged, each product including a first portion provided with information from which the product can be identified, and a second portion provided with one piece of information among a plurality of predetermined variations. This shelf state detection method includes: acquiring a shelf photographed image obtained by photographing a shelf on which a plurality of products are disposed; detecting, as a common region, the second portion of each product from the shelf photographed image; and, if the common region is not included in an object region in which an object should be present in the shelf photographed image, determining that the product is out of stock in the object region.

Description

Shelf state detection apparatus and shelf state detection method

[0001] The present disclosure relates to a shelf state detection apparatus and a shelf state detection method.

[0002] Conventionally, as disclosed in Patent Document 1, there is known a technique of detecting a product out-of-stock by analyzing a captured image obtained by photographing products displayed on a shelf.

[0003] Japanese Unexamined Patent Publication No. 2017-157032

[0004] However, depending on the characteristics of the shelf or the product, conventional analysis of captured images may not be able to perform product out-of-stock detection and / or product identification with sufficient accuracy.

[0005] An object of the present disclosure is to provide a technique capable of performing product out-of-stock detection and / or product identification with sufficient accuracy in accordance with the characteristics of a shelf or products.

[0006] One aspect of the present disclosure is a shelf state detection apparatus that detects the state of a shelf on which a plurality of products are placed, wherein the product includes a first portion to which information capable of identifying the product is attached, and a second portion to which one of a plurality of predetermined variations of information is attached, the shelf state detection apparatus including: an acquisition unit that acquires a shelf captured image obtained by photographing the shelf on which the plurality of products are placed; a common region detection unit that detects the second portion of each product as a common region from the shelf captured image; and an out-of-stock detection unit that determines that a product is out of stock in an object region where an object should be present in the shelf captured image when the common region is not included in the object region.

[0007] One aspect of the present disclosure is a shelf state detection method for detecting the state of a shelf on which a plurality of products are placed, wherein the product includes a first portion to which information capable of identifying the product is attached, and a second portion to which one of a plurality of predetermined variations of information is attached, the shelf state detection method includes: acquiring a shelf captured image obtained by photographing the shelf on which the plurality of products are placed; detecting the second portion of each product as a common region from the shelf captured image; and determining that a product is out of stock in an object region where an object should be present in the shelf captured image when the common region is not included in the object region.

[0008] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0009] According to this disclosure, product shortages can be detected and / or product identification can be performed with sufficient accuracy depending on the characteristics of the shelf or product.

[0010] A diagram showing an example of a shelf displaying multiple cigarettes. A diagram showing an example of cigarette packaging. A block diagram showing an example of the configuration of the shelf state detection system according to Embodiment 1. A flowchart showing an example of a stockout detection method according to Embodiment 1. A diagram explaining the case in which a stockout is determined according to Embodiment 1. A diagram explaining the case in which a stockout is not determined according to Embodiment 1. A schematic diagram showing an example of displaying a stockout area according to Embodiment 1. A block diagram showing an example of the configuration of the shelf state detection system according to Embodiment 2. A flowchart showing an example of a product identification method according to Embodiment 2. A diagram showing an example of the configuration of the shelf state list according to Embodiment 2. A flowchart showing an example of an identification area extraction method according to Embodiment 2. A diagram explaining the coordinate and dimensional information of the object area Di, common area Cj, and identification area Ei according to Embodiment 2. A diagram explaining the arrangement type of the object area Di and common area Cj according to Embodiment 2. A schematic diagram showing an example of displaying product identification according to Embodiment 2. A block diagram showing an example of the hardware configuration of the shelf state detection device according to this disclosure.

[0011] Embodiments of the present disclosure will be described in detail below, with appropriate reference to the drawings. However, descriptions that are unnecessarily detailed may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid the following description becoming unnecessarily verbose and to facilitate understanding for those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter of the claims. The functions of one configuration shown in this embodiment may be realized by two or more physical configurations, or the functions of two or more configurations may be realized by, for example, one physical configuration.

[0012] (Background to this disclosure) Figure 1 shows an example of a shelf 1 on which multiple cigarettes 2 are displayed. Figure 2 shows an example of a cigarette package 2.

[0013] As shown in Figures 1 and 2, in several countries around the world (e.g., Europe, Thailand, Austria, etc.), it is stipulated that harmful warnings (photographs, illustrations, and / or text, etc., hereinafter referred to as harmful warnings 3) must be added (e.g., printed or displayed) to cigarette packages 2. There are multiple patterns of harmful warnings 3, and it is stipulated that cigarette manufacturers 2 randomly add one of these patterns of harmful warnings 3 to their cigarette packages 2. In other words, even cigarette packages 2 with the same product name 4 may have different patterns of harmful warnings 3, or cigarette packages 2 with different product names 4 may have the same pattern of harmful warnings 3.

[0014] Furthermore, as shown in Figures 2(a), (b), and (c), the area to which the harmful substance warning 3 is applied is specified to be a predetermined percentage or more of the area of ​​the cigarette 2 package (for example, 65% or more), but the position on the cigarette 2 package where the harmful substance warning 3 is applied is not specified and is arbitrary.

[0015] Therefore, it is difficult to train a learning model to recognize the package design of cigarette 2 and its product name. This is because even cigarettes with the same product name 4 may have different patterns of harmful warnings 3, and cigarettes with different product names 4 may have the same pattern of harmful warnings 3. As a result, there is no correspondence (correlation) between the features of the harmful warning 3, which occupies the majority of the package, and the product name. Generally, learning models learn the image that occupies the majority of the package as the main feature of the product. In the case of cigarettes 2, this means that the harmful warning 3, which has no correlation with individual products, is learned as a feature of each product. Consequently, a learning model that has learned the package design of cigarette 2 will mistakenly identify another cigarette with the same harmful warning 3 as cigarette 2. In other words, it is difficult to generate a learning model to identify the product name 4 of each cigarette 2 from shelf images 100 using conventional methods.

[0016] Another method for detecting a shortage of cigarettes 2 without identifying the cigarettes 2 themselves is to determine that a shortage has occurred when it is detected that the item is not present in the area where cigarettes 2 should be. With this method, it is possible to determine whether or not a shortage has occurred regardless of the design of cigarettes 2, and therefore, the presence of the harmful substance warning 3 will not cause a misjudgment of whether or not a shortage has occurred. However, as shown in the shortage area 7 in Figure 1, if a cigarette 2 is missing, the stand that supported the cigarette 2 (hereinafter referred to as the support stand 5) is exposed, making it difficult to detect the shortage using this method. When a cigarette 2 is missing, the support stand 5 is captured in the shortage area 7 of cigarette 2 in the shelf image 100.

[0017] Therefore, when an object detection model capable of detecting objects in an image is used to perform object detection on the missing area 7 of cigarette 2, the object detection model does not fail to detect an object, but rather detects the support stand 5. As a result, because an object (support stand 5) is detected in the missing area 7, it is determined that cigarette 2 is not missing. The design of the support stand 5 is arbitrary; for example, it may be plain, or it may have an illustration or handwritten text by a store clerk. Therefore, it is difficult to train the object detection model to recognize designs that should not be detected as objects.

[0018] Therefore, this disclosure describes a shelf state detection system 10, a shelf state detection device 20, and a shelf state detection method, etc., which use a shelf image 100 taken of a shelf 1 on which multiple cigarettes 2 bearing a harmful warning 3 as shown in Figure 1 are arranged (displayed) to detect shortages of cigarettes 2 and / or identify the product name 4 of the cigarettes 2.

[0019] While this disclosure uses tobacco as an example of a product or article, the content of this disclosure is not limited to tobacco and is applicable to various products or articles that are randomly assigned multiple design patterns.

[0020] (Embodiment 1) <System Configuration> Figure 3 is a block diagram showing an example of the configuration of the shelf state detection system 10 according to Embodiment 1.

[0021] The shelf status detection system 10 comprises a camera 11, a display device 12, and a shelf status detection device 20. The camera 11, the display device 12, and the shelf status detection device 20 may transmit and receive data through a predetermined communication network 13.

[0022] The imaging device 11 photographs a shelf 1 on which multiple cigarettes 2 are displayed, as shown in Figure 1, and generates a shelf image 100. The display device 12 displays information and images. The imaging device 11 and the display device 12 may be separate devices, such as a surveillance camera and a PC display, or they may be an integrated device, such as a smartphone or tablet terminal with a camera.

[0023] The shelf state detection device 20 is a device that detects the display state of cigarettes 2 on shelf 1 using shelf images 100. Embodiment 1 describes how the shelf state detection device 20 detects areas on shelf 1 where cigarettes 2 are missing (missing area 7).

[0024] The shelf status detection device 20 includes, as functions, an acquisition unit 21, an object detection unit 22, a common area detection unit 23, a missing item detection unit 24, and an output unit 25. These functions may be realized by the processor 1001 shown in Figure 15 working in cooperation with the memory 1002 to execute a program.

[0025] The acquisition unit 21 acquires the shelf image 100 from the camera 11 via the communication network 13, etc.

[0026] The object detection unit 22 detects objects from the shelf image 100 using known techniques for detecting objects from images. As a known technique for detecting objects from images, an object detection model configured with deep learning may be used. The objects detected here may include cigarettes 2 and the support stand 5 that is exposed when the cigarettes 2 are out of stock. Hereinafter, the region of objects detected from the shelf image 100 will be referred to as the object region 110.

[0027] The common area detection unit 23 uses a pre-trained model (detection model) to detect the area of ​​the harmful warning 3 on the cigarette package 2, and detects the area of ​​the harmful warning 3 from the shelf image 100. The training model may be composed of deep learning. Hereinafter, the area of ​​the harmful warning 3 detected from the shelf image 100 will be referred to as the common area 120.

[0028] The learning model (detection model) may be trained using a low-resolution image of the cigarette 2 package. The common area detection unit 23 may then reduce the resolution of the shelf image 100 and input the reduced-resolution shelf image 100 into the learning model (detection model) to detect the common area 120. In this way, by using a low-resolution image, even if the text and / or picture of the harmful warning 3 are updated, the common area detection unit 23 can detect the common area 120 using the learning model without performing additional training on the learning model (detection model). However, using a low-resolution image may result in the loss of information specific to each variation of the harmful warning 3. However, in this embodiment, it is sufficient to detect that the harmful warning 3 exists, and it is not necessary to identify variations of the harmful warning 3. Therefore, in the configuration of this embodiment, there is no disadvantage to using a low-resolution image for training the learning model that detects the area of ​​the harmful warning 3.

[0029] The missing item detection unit 24 uses the object region 110 detected by the object detection unit 22 and the common region 120 detected by the common region detection unit 23 to detect whether or not cigarettes 2 are missing from the object region 110. Details of this missing item detection method will be described later. In the shelf image 100, the object region 110 that is detected as missing is the missing item region 7.

[0030] The output unit 25 displays the missing items area 7 in the shelf image 100 on the display device 12 in a manner that makes them visible. Details of the display method will be described later.

[0031] <Method for detecting missing items> Figure 4 is a flowchart showing an example of a method for detecting missing items according to Embodiment 1. Figure 5 is a diagram illustrating the case in which it is determined that an item is missing according to Embodiment 1. Figure 6 is a diagram illustrating the case in which it is determined that an item is not missing according to Embodiment 1. Next, with reference to Figures 4, 5, and 6, a method by which the shelf state detection device 20 detects missing item areas 7 from the shelf image 100 will be described.

[0032] The acquisition unit 21 acquires the shelf image 100 from the imaging device 11 (S101).

[0033] The object detection unit 22 detects the region where an object exists from the shelf image 100 as the object region 110 (S102). As described above, the objects detected here may include the cigarettes 2 and the support stand 5.

[0034] The object detection unit 22 sorts the multiple object regions 110 detected in step S102 and generates an object region list 210 (S103). For example, the object detection unit 22 sets the coordinates of the top left of the shelf image 100 to (X, Y) = (0, 0) and sorts the multiple object regions 110 in the following steps (A1) to (A3).

[0035] (A1) The object detection unit 22 refers to the XY coordinate points of each of the multiple object regions 110 detected and sorts the multiple object regions 110 in descending order of Y coordinate. (A2) For the multiple object regions 110 sorted in (A1), the object detection unit 22 repeatedly assigns the object regions 110 located on the first shelf from the top of shelf 1, starting with the one with the smallest Y coordinate and up to a threshold N, and the object regions 110 located on the second shelf from the top of shelf 1, starting with the next one within threshold N and up to another threshold N, thereby dividing the multiple object regions 110 into shelf levels. N is a value corresponding to the height of one shelf level. (A3) For each shelf level divided in (A2), the object detection unit 22 sorts the multiple object regions 110 in descending order of X coordinate.

[0036] Hereafter, the sorted object regions 110 included in the object region list 210 will be referred to as object region Di. i is an integer such as 0, 1, 2, ... In other words, i indicates the number of object region 110.

[0037] Meanwhile, the common area detection unit 23 detects the area of ​​the harmful warning 3 as the common area 120 from the shelf image 100 (S104).

[0038] The common area detection unit 23 sorts the multiple common areas 120 detected in step S104 and generates a common area list 220 (S105). For example, the common area detection unit 23 sorts the multiple common areas 120 in the same steps as (A1) to (A3) described above.

[0039] Hereinafter, the sorted common regions 120 included in the common region list 220 are sequentially expressed as common regions Cj. j is an integer such as 0, 1, 2, .... In other words, j indicates the number of the common region.

[0040] Note that, although FIG. 4 shows an example in which steps S102 and S103, and steps S104 and S105 are executed in parallel, these processes may be executed in series.

[0041] The out-of-stock detection unit 24 assigns 0 to a variable i and assigns 0 to a variable j (S106). In other words, the out-of-stock detection unit 24 initializes the variables i and j.

[0042] The out-of-stock detection unit 24 determines whether or not the common region Cj is included in the object region Di (S107). For example, when the common region Cj and the object region Di have a positional relationship as shown in FIG. 5, the out-of-stock detection unit 24 determines that the common region Cj is not included in the object region Di. For example, when the common region Cj and the object region Di+1 have a positional relationship as shown in FIG. 6, the out-of-stock detection unit 24 determines that the common region Cj is included in the object region Di+1.

[0043] When the out-of-stock detection unit 24 determines that the common region Cj is not included in the object region Di (S107: NO), as shown in FIG. 5, the out-of-stock detection unit 24 associates out-of-stock information indicating that the product is out of stock with the object region number i of the object region Di in the shelf state list 230 (S108). Then, the process proceeds to step S111.

[0044] When the out-of-stock detection unit 24 determines that the common region Cj is included in the object region Di (S107: YES), the out-of-stock detection unit 24 associates presence information indicating that the product is not out of stock with the object region number i of the object region Di in the shelf state list 230 (S109).

[0045] The out-of-stock detection unit 24 adds 1 (increments) to the variable j (S110).

[0046] The out-of-stock detection unit 24 adds 1 (increments) to the variable i (S111).

[0047] The out-of-stock detection unit 24 determines whether or not the current object region Di is the last one in the object region list 210 (S112).

[0048] If the current object region Di is not the last one in the object region list 210 (S112: NO), the process returns to step S107.

[0049] If the current object region Di is the last one in the object region list 210 (S112: YES), the present process ends.

[0050] Through the above processing, a shelf state list 230 is generated, in which out-of-stock information is associated with an object region Di where the cigarette 2 is out of stock, and presence information is associated with an object region Di where the cigarette 2 is not out of stock.

[0051] Therefore, the shelf state detection device 20 according to the first embodiment can accurately detect the out-of-stock region 7 of the cigarettes 2 from the shelf-captured image 100 where a plurality of cigarettes 2 each provided with a hazard warning 3 as shown in FIG. 1 and FIG. 2 are displayed.

[0052] If an attempt is made to detect the presence or absence of cigarettes 2 from the shelf-captured image 100 only by object detection according to the conventional technology, it is difficult to accurately detect cigarettes 2 provided with hazard warnings 3 of different patterns as objects, and furthermore, the support base 5 exposed when a cigarette 2 is out of stock is also detected as an object. For this reason, it is difficult for the conventional technology to accurately detect out-of-stock cigarettes 2 from the shelf-captured image 100.

[0053] In contrast, according to the method according to the first embodiment described with reference to FIG. 4, FIG. 5 and FIG. 6, by combining the detection result of the object region 110 and the detection result of the common region 120, as described above, the out-of-stock region 7 can be accurately detected from the shelf-captured image 100.

[0054] <Display Example> FIG. 7 is a schematic diagram showing a display example of the out-of-stock region 7 according to the first embodiment.

[0055] As shown in Figure 7, the output unit 25 overlays a missing item image 301 (for example, a predetermined frame and / or characters indicating missing items) on the object region Dm (i.e., missing item region 7) in the shelf state list 230 that is associated with missing item information in the shelf state list 230 of the shelf image 100. m is an integer from 0, 1, 2, ... This allows the user to easily recognize the location on shelf 1 where cigarettes 2 are missing.

[0056] Furthermore, if the shelf state detection device 20 has a shelf layout plan (also called a planogram) that indicates which product name of cigarette 2 should be placed in which location on shelf 1, it may perform the following processing.

[0057] The output unit 25 overlays the product name 302 of the cigarettes 2 that should be placed in the object region Dm (i.e., the missing area 7) associated with the missing item information in the shelf image 100, as indicated by the shelf layout plan. This allows the user to easily recognize the product name of the cigarettes 2 that should be placed in the location on shelf 1 where cigarettes 2 are missing.

[0058] Furthermore, the output unit 25 superimposes the product name 302 of the cigarettes 2 that should be placed in the object region Dn, as indicated by the shelf layout plan, onto the object region Dn associated with the existence information in the shelf image 100. At this time, the output unit 25 may superimpose the product name 302 within the common region 120 included in the object region Dn.

[0059] This allows users to easily verify whether cigarettes 2 are arranged on shelf 1 according to the shelf layout plan by comparing the product name 4, which is actually written in the area where the harmfulness warning 3 for cigarettes 2 is not attached, with the product name 302, which is superimposed in the common area 120.

[0060] Furthermore, if the product name of the cigarette 2 located in the object region 110 is displayed above (or below) the outside of the object region 110, it becomes difficult to understand because it overlaps with the display of the product name of the adjacent cigarette 2. However, by displaying the product name 302 within the common region 120 as in this embodiment, this problem is avoided.

[0061] (Embodiment 2) In Embodiment 2, in addition to the stockout detection described in Embodiment 1, a method for identifying the product name 4 of each cigarette 2 placed on the shelf 1 will be described. In Embodiment 2, components already described in Embodiment 1 will be given a common reference number and their description may be omitted.

[0062] Figure 8 is a block diagram showing an example of the configuration of the shelf state detection system 10 according to Embodiment 2.

[0063] The shelf status detection device 20 includes, as functions, an acquisition unit 21, an object detection unit 22, a common area detection unit 23, a missing item detection unit 24, a product identification unit 26, and an output unit 25.

[0064] The acquisition unit 21, object detection unit 22, common area detection unit 23, and missing item detection unit 24 are the same as in Embodiment 1, so their description will be omitted.

[0065] The product identification unit 26 identifies the product name 4 of the cigarettes 2 located in the object region 110 using the object region 110 detected by the object detection unit 22, the common region 120 detected by the common region detection unit 23, and the identification region 130, which is an area within the object region 110 other than the common region 120. The product identification unit 26 uses a trained model (identification model) that has been trained to identify the product name 4 of the cigarettes 2 that is attached to an area of ​​the cigarette package other than the harmful warning 3, to identify the product name 4 of the cigarettes 2 located in the object region 110 from the image of the identification region 130 within the object region 110 in the shelf image 100. Details of how the product identification unit 26 extracts the identification region 130 from the object region 110 will be described later.

[0066] <Product Identification Method> Figure 9 is a flowchart showing an example of a product identification method according to Embodiment 2. Figure 10 is a diagram showing an example of the configuration of a shelf state list 230 according to Embodiment 2. Next, with reference to Figures 9 and 10, a method by which the shelf state detection device 20 identifies the product name 4 of each cigarette 2 displayed on the shelf 1 from the shelf image 100 will be described.

[0067] The acquisition unit 21 acquires the shelf image 100 from the imaging device 11 (S201).

[0068] The object detection unit 22 detects the region where an object exists from the shelf image 100 as the object region 110, similar to step S102 in Figure 4 (S202).

[0069] The object detection unit 22 sorts the multiple object regions 110 detected in step S202, similar to step S103 in Figure 4, and generates an object region list 210 (S203).

[0070] Meanwhile, the common area detection unit 23 detects the area of ​​the hazardous material warning 3 as the common area 120 from the shelf image 100, similar to step S104 in Figure 4 (S204).

[0071] The common area detection unit 23 sorts the multiple common areas 120 detected in step S204, similar to step S105 in Figure 4, and generates a common area list 220 (S205).

[0072] The missing item detection unit 24 assigns 0 to variable i and 0 to variable j (S206). In other words, the missing item detection unit 24 initializes variables i and j.

[0073] The missing item detection unit 24 determines whether the common area Cj is included in the object area Di, similar to step S107 in Figure 4 (S207).

[0074] If the missing item detection unit 24 determines that the common area Cj is not included in the object area Di (S207: NO), it associates the missing item information with the object area number i of the object area Di in the shelf status list 230, similar to step S108 in Figure 4 (S208). Then the process proceeds to step S213.

[0075] If the missing item detection unit 24 determines that the common area Cj is included in the object area Di (S207: YES), it associates the existence information with the object area number i of the object area Di in the shelf state list 230, similar to step S109 in Figure 4 (S209).

[0076] The product identification unit 26 extracts the identification region Ei from the object region Di (S210). Details of the method for extracting the identification region 130 will be described later (see Figure 12).

[0077] The product identification unit 26 analyzes the image within the extracted identification area Ei to identify the product name 4 of the cigarette 2 located in the object area Di (S211). Then, as shown in Figure 10, the product identification unit 26 associates the identified product name 4 of the cigarette 2 with the object area number i in the shelf state list 230.

[0078] The missing item detection unit 24 adds (increments) 1 to the variable j (S212).

[0079] The missing item detection unit adds (increments) 1 to the variable i (S213).

[0080] The missing item detection unit 24 determines whether the current object region Di is the last item in the object region list 210 (S214).

[0081] If the current object region Di is not the last in the object region list 210 (S214: NO), the process returns to step S207.

[0082] If the current object region Di is the last in the object region list 210 (S214: YES), this process terminates.

[0083] Through the above process, a shelf status list 230 is generated in which the shortage information is associated with the object area Di where cigarette 2 is missing, and the product name 4 of cigarette 2 is associated with the object area Di where cigarette 2 is not missing.

[0084] Therefore, the shelf condition detection device 20 according to Embodiment 2 can accurately detect the areas where cigarettes 2 are out of stock 7 and the product names 4 of the cigarettes 2 from a shelf image 100 in which multiple cigarettes 2 with harmful warnings 3, as shown in Figures 1 and 2, are displayed.

[0085] <Identification Region Extraction Method> Figure 11 is a flowchart showing an example of the identification region extraction method according to Embodiment 2. Figure 12 is a diagram illustrating the coordinate and dimensional information of the object region Di, common region Cj, and identification region Ei according to Embodiment 2. Figure 13 is a diagram illustrating the arrangement type of the object region Di and common region Cj according to Embodiment 2. The process of step S210 in Figure 9 will be described in detail below with reference to Figures 11 to 13.

[0086] The product identification unit 26 obtains the coordinates of the top left (Dxi, Dyi), width (Dwi), and height (Dhi) of the object region Di shown in Figure 12, and obtains the coordinates of the top left (Cxj, Cyj), width (Cwj), and height (Chj) of the common region Cj (S301).

[0087] The product identification unit 26 determines whether (Dhi - Chj) ≥ (Dwi - Cwj) (S302). In other words, the product identification unit 26 determines whether the arrangement type of the object region Di and the common region Cj is type A, where the common region Cj is located at the top or bottom of the object region Di, or type B, where the common region Cj is located at the left or right of the object region Di, as shown in Figure 13.

[0088] First, we will explain the case where the determination result of step S302 is (Dhi - Chj) ≥ (Dwi - Cwj) (S302: YES), that is, the case of type A as shown in Figure 13.

[0089] The product identification unit 26 determines whether (Dyi + Dhi / 2) ≥ (Cyj + Chj / 2) (S303). In other words, the product identification unit 26 determines whether the common area Cj is located at the top or bottom of the object area Di in type A shown in Figure 13.

[0090] If (Dyi + Dhi / 2) ≥ (Cyj + Chj / 2) (S303: YES), then, as shown in type A-1 of Figure 13, the common area Cj is located at the top of the object area Di, and as a result, the identification area Ei is located at the bottom of the object area Di. Therefore, the product identification unit 26 generates identification area information where the coordinates of the top left of the identification area Ei (Exi, Eyi) are (Dxi, Dyi - Chj), the width of the identification area Ei (Ew) is (Dwi), and the height of the identification area Ei (Eh) is (Dhi - Chj), and associates this with the object area number i in the shelf state list 230 shown in Figure 10 (S304). Then, this process is completed.

[0091] If (Dyi + Dhi / 2) < (Cyj + Chj / 2) (S303: NO), then, as shown in type A-2 of Figure 13, the common area Cj is located at the bottom of the object area Di, and as a result, the identification area Ei is located at the top of the object area Di. Therefore, the product identification unit 26 generates identification area information where the coordinates of the top left of the identification area Ei (Exi, Eyi) are (Dxi, Dyi), the width of the identification area Ei (Ew) is (Dwi), and the height of the identification area Ei (Eh) is (Dhi - Chj), and associates this with the object area number i in the shelf state list 230 shown in Figure 10 (S305). Then, this process is completed.

[0092] Next, we will explain the case where the determination result of step S302 is (Dhi-Chj) < (Dwi-Cwj) (S302: NO), that is, the case of type B shown in Figure 13.

[0093] The product identification unit 26 determines whether (Dxi + Dwi / 2) ≥ (Cxj + Cwj / 2) (S310). In other words, in type B, the product identification unit 26 determines whether the common area Cj is located on the left or right side of the object area Di.

[0094] If (Dxi + Dwi / 2) ≥ (Cxj + Cwj / 2) (S310: YES), then, as shown in type B-1 of Figure 13, the common area Cj is located on the left side of the object area Di, and as a result, the identification area Ei is located on the right side of the object area Di. Therefore, the product identification unit 26 generates identification area information where the coordinates of the top left of the identification area Ei (Exi, Eyi) are (Dxi + Cwj, Dyi), the width of the identification area Ei (Ew) is (Dwi - Cwj), and the height of the identification area Ei (Eh) is (Dhi), and associates this with the object area number i in the shelf state list 230 shown in Figure 10 (S311). Then, this process is completed.

[0095] If (Dxi + Dwi / 2) < (Cxj + Cwj / 2) (S310: NO), then, as shown in type B-2 of Figure 13, the common area Cj is located in the right part of the object area Di, and as a result, the identification area Ei is located in the left part of the object area Di. Therefore, the product identification unit 26 generates identification area information where the coordinates of the upper left of the identification area Ei (Exi, Eyi) are (Dxi, Dyi), the width of the identification area Ei (Ew) is (Dwi - Cwj), and the height of the identification area Ei (Eh) is (Dhi), and associates this with the object area number i in the shelf state list 230 shown in Figure 10 (S312). Then, this process is completed.

[0096] Through the above processing, the portion of the identification region Ei within the object region Di can be identified. Therefore, in step S211 in Figure 9, the product identification unit 26 can analyze the image within the identified identification region Ei to identify the product name 4 of the cigarette 2.

[0097] <Example of display> Figure 14 is a schematic diagram showing an example of a product identification display according to Embodiment 2.

[0098] The shelf state detection device 20 may have a shelf layout plan (also called a planogram) that indicates which product name of cigarette 2 should be placed in which location on shelf 1.

[0099] As shown in Figure 12, the output unit 25 may superimpose the following onto the shelf image 100 and display them on the display device 12: the product name of the cigarettes 2 to be placed in the object area 110, obtained from the shelf allocation plan (hereinafter referred to as shelf allocation product name 311), and the product name obtained from the image analysis of the identification area 130 within the object area 110, shelf allocation plan (hereinafter referred to as identification product name 312), within the range of the common area 120 within the object area 110 to which the product name is associated in the shelf status list 230.

[0100] This allows users to easily verify whether cigarette packs 2 are arranged according to the planogram (shelf layout plan).

[0101] Furthermore, by displaying the shelf allocation product name 311 and the identification product name 312 within the common area 120 (i.e., the area of ​​the harmful product warning 3), users can easily compare the part of the cigarette 2 package where the product name 4 is written (i.e., the image of the identification area 130) with the shelf allocation product name 311 and the identification product name 312 to confirm whether the identification product name 312 is incorrect, or whether the placement of the cigarette 2 is incorrect, etc. For example, in Figure 14, the product name 4 on the cigarette 2 package is "AAA", the shelf allocation product name 311 is "AAA", and the identification product name 312 is "BBB", so the cigarette 2 is placed according to the shelf allocation plan and the user can easily confirm that the identification product name 312 is incorrect.

[0102] (Hardware Configuration) Figure 15 is a block diagram showing an example of the hardware configuration of the shelf state detection device 20 according to this disclosure.

[0103] The shelf status detection device 20 comprises, as hardware, a processor 1001, memory 1002, storage 1003, input device 1004, display device 1005, and communication device 1006. The shelf status detection device 20 may be interpreted as a computer, information processing device, or server device.

[0104] The processor 1001 reads a predetermined program from the memory 1002 and executes it to realize the functions of the shelf state detection device 20 described above. For example, the processing of the acquisition unit 21, object detection unit 22, common area detection unit 23, missing item detection unit 24, product identification unit 26, and output unit 25 described above may be performed by the processor 1001. The processor 1001 may also be read as a Central Processing Unit (CPU), controller, control device, Large Scale Integration (LSI), etc. Furthermore, the processor 1001 may include a Graphics Processing Unit (GPU) and / or a Neural Network Processing Unit (NPU).

[0105] The memory 1002 stores programs and data for realizing the functions of the shelf state detection device 20 described above. The memory 1002 may be composed of a volatile storage medium and / or a non-volatile storage medium.

[0106] The storage device 1003 stores programs and data for realizing the functions of the shelf state detection device 20 described above. The storage device 1003 may be made of a non-volatile storage medium. Examples of the storage device 1003 include Hard Disk Drives (HDDs), Solid State Drives (SSDs), and flash memory.

[0107] For example, the object area list 210, common area list 220, and shelf status list 230 mentioned above may be stored in memory 1002 and / or storage 1003.

[0108] The input device 1004 is a device that receives input from the user. Examples of the input device 1004 include a keyboard, mouse, touchpad, touch panel, and microphone.

[0109] The display device 1005 is a device that displays characters, images, etc. Examples of the display device 1005 include liquid crystal displays and organic EL displays.

[0110] The communication device 1006 is a device that connects the shelf status detection device 20 to the communication network 13. Examples of the communication network 13 include wired LAN (e.g., Ethernet®), wireless LAN (e.g., Wi-Fi®), the Internet, mobile communication networks (e.g., 4G, 5G), and Bluetooth®. As described above, the shelf image 100 captured and generated by the imaging device 11 may be received by the communication device 1006 via the communication network 13.

[0111] (Other variations) In the above-described embodiment, an example was explained in which the hazard warning 3 is used as the common area. However, the common area may be any other type of display, as long as it is shared across multiple products and has multiple predetermined variations.

[0112] In the embodiment described above, the object detection unit 22 detected the object region 110. However, other methods may be used to detect the object region 110. For example, if the position and size of the area where cigarettes 3 are placed on the shelf 1 are predetermined, the processing by the object detection unit 22 may be omitted, and each of the predetermined areas may be detected as an object region 110.

[0113] (Summary of this disclosure) Based on the description of Embodiment 1 above, the following technology is disclosed.

[0114] <Technology 1> A shelf state detection device (20) according to one embodiment for detecting the state of a shelf (1) on which multiple products are arranged, wherein each product includes a first part to which information that can identify the product is attached, and a second part to which information that is attached to one of a predetermined number of variations is attached, and comprises: an acquisition unit (21) that acquires a shelf image (100) taken of the shelf on which the multiple products are arranged; a common area detection unit (23) that detects the second part of each product as a common area (120) from the shelf image; and a shortage detection unit (24) that determines that a product is missing in an object area (110) where an object should be present in the shelf image if the common area is not included in the object area. This makes it possible to detect product shortages with sufficient accuracy on a shelf on which multiple products are arranged, each product including a first part to which information that can identify the product is attached, and a second part to which information that is attached to one of a predetermined number of variations is attached.

[0115] <Technology 2> In the shelf state detection device (20) described in Technology 1, the missing item detection unit determines that there are no missing items in the object area if the common area is included in the object area. This makes it possible to detect whether or not an item is missing with sufficient accuracy.

[0116] <Technology 3> The shelf status detection device described in Technology 2 further includes a product identification unit (26) that, when the missing item detection unit determines that there are no missing items in the object area, extracts the area of ​​the object area of ​​the shelf image excluding the common area as an identification area (130), and analyzes the image in the extracted identification area to identify product information. This makes it possible to detect product information of items placed on the shelf with sufficient accuracy.

[0117] <Technology 4> In the shelf state detection device described in any one of Technologies 1 to 3, the common area detection unit reduces the resolution of the shelf image and detects the second portion of each product as a common area from the reduced-resolution shelf image. As a result, even if the variation of the second portion is updated, the second portion of the product can be detected as a common area without updating the common area detection unit.

[0118] <Technology 5> The shelf status detection device described in any one of Technologies 1 to 4 further comprises an output unit that outputs an image in which information indicating a missing item is superimposed on the area of ​​an object that the missing item detection unit has determined to be missing in the shelf image. This allows the user to easily recognize the location of missing items on the shelf.

[0119] <Technology 6> The shelf status detection device according to any one of Technologies 1 to 5 further comprises an output unit that outputs an image in which the product name to be placed in the object area of ​​the shelf that the missing item detection unit has determined to be missing in the shelf image is superimposed. This allows the user to easily recognize the product name to be placed in the position where the product is missing on the shelf.

[0120] <Technology 7> The shelf status detection device according to any one of technologies 3 to 6 further comprises an output unit that outputs an image in which the product name identified by the product identification unit is superimposed on the common area within the object area in the shelf image that the missing item detection unit has determined to be not missing. This allows the user to confirm whether the shelf status detection device has correctly identified the product names of the products placed on the shelf.

[0121] <Technology 8> In the shelf state detection device described in Technology 7, the output unit outputs an image in which the product name to be placed in the object area of ​​the shelf is further superimposed on the common area within the object area that the missing item detection unit has determined to be not missing in the shelf image. This allows the user to confirm whether the products placed on the shelf are arranged according to the shelf layout plan (planogram).

[0122] <Technology 9> The shelf state detection device described in any one of Technologies 1 to 8 further comprises an object detection unit that detects an area where an object exists from the shelf image and sets the detected area as the object area. This makes it possible to detect with sufficient accuracy whether or not a product is out of stock, even if an object other than the product (e.g., a support stand 5) is detected when the product (e.g., cigarettes 2) is out of stock.

[0123] <Technology 10> In the shelf condition detection device described in any one of Technologies 1 to 9, the product is cigarettes, the first part is provided with information indicating the product name of the cigarettes, and the second part is provided with one of a predetermined set of variations of harmful warnings (3). This makes it possible to detect cigarette shortages with sufficient accuracy on a shelf (1) where multiple cigarettes are placed, including a first part (identification area 130) provided with information that allows identification of cigarettes and a second part (common area 120) provided with one of a predetermined set of variations of harmful warnings (3).

[0124] <Technical 11> A shelf state detection method according to one embodiment for detecting the state of a shelf (1) on which multiple products are arranged, wherein the product includes a first part to which information that can identify the product is attached, and a second part to which information that is attached to one of a predetermined number of variations is attached, a shelf image (100) of the shelf on which the multiple products are arranged is acquired, the second part of each product is detected as a common area (120) from the shelf image, and if the common area is not included in the object area (110) where an object should be present in the shelf image, it is determined that the product is out of stock in that object area. As a result, it is possible to detect product shortages with sufficient accuracy on a shelf on which multiple products are arranged, each product including a first part to which information that can identify the product is attached, and a second part to which information that is attached to one of a predetermined number of variations is attached.

[0125] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention.

[0126] The technology disclosed herein is useful for detecting missing items and identifying products among multiple items placed on a shelf.

[0127] 1 Shelf 2 Cigarettes 3 Harmful Warning 4 Product Name 5 Support Stand 7 Out-of-Stock Area 10 Shelf Status Detection System 11 Shooting Device 12 Display Device 13 Communication Network 20 Shelf Status Detection Device 21 Acquisition Unit 22 Object Detection Unit 23 Common Area Detection Unit 24 Out-of-Stock Detection Unit 25 Output Unit 26 Product Identification Unit 100 Shelf Image 110 Object Area 120 Common Area 130 Identification Area 210 Object Area List 220 Common Area List 230 Shelf Status List 1001 Processor 1002 Memory 1003 Storage 1004 Input Device 1005 Display Device 1006 Communication Device

Claims

1. A shelf state detection device for detecting the state of a shelf on which multiple products are arranged, wherein each product includes a first part to which information that can identify the product is attached, and a second part to which information that corresponds to one of a predetermined number of variations is attached, the device comprises: an acquisition unit for acquiring a shelf image taken of the shelf on which the multiple products are arranged; a common area detection unit for detecting the second part of each product as a common area from the shelf image; and a missing item detection unit for determining that a product is missing in an object area if the common area is not included in an object area where an object should exist in the shelf image.

2. The shelf status detection device according to claim 1, wherein the missing item detection unit determines that there are no missing items in the object area if the object area includes the common area.

3. The shelf status detection device according to claim 2, further comprising:

3. When the missing item detection unit determines that there are no missing items in the object area, the device extracts an area in the object area of ​​the shelf image excluding the common area as an identification area, and analyzes the image in the extracted identification area to identify product information.

4. The shelf condition detection device according to claim 1, wherein the common area detection unit reduces the resolution of the shelf image and detects the second portion of each product as a common area from the reduced-resolution shelf image.

5. The shelf condition detection device according to claim 1, further comprising: an output unit that outputs an image in which information indicating a missing item is superimposed on the area of ​​an object that the missing item detection unit has determined to be missing in the shelf image.

6. The shelf state detection device according to claim 1, further comprising: an output unit that outputs an image in which the product name to be placed on the shelf is superimposed on the object region in the shelf image that the missing item detection unit has determined to be missing.

7. The shelf condition detection device according to claim 3, further comprising: an output unit that outputs an image in which the product name identified by the product identification unit is superimposed on the common area within the area of ​​an object that the missing item detection unit has determined to be not missing in the shelf image.

8. The shelf state detection device according to claim 7, wherein the output unit outputs an image in which the product name to be placed in the object area of ​​the shelf is further superimposed on the common area in the object area in the shelf image that the missing item detection unit has determined to be not missing.

9. The shelf state detection device according to claim 1, further comprising an object detection unit that detects an area where an object exists from the shelf image and sets the detected area as the object area.

10. The shelf condition detection device according to any one of claims 1 to 9, wherein the product is tobacco, the first part is provided with information indicating the product name of the tobacco, and the second part is provided with one of a predetermined set of variations of harmful warnings.

11. A shelf state detection method for detecting the state of a shelf on which multiple products are arranged, wherein each product includes a first part to which information that can identify the product is attached, and a second part to which information that corresponds to one of a predetermined number of variations is attached, a shelf image is taken of the shelf on which the multiple products are arranged, the second part of each product is detected as a common area from the shelf image, and if the common area is not included in the object area where an object should be present in the shelf image, it is determined that the product is out of stock in that object area.

Images