Product out-of-stock detection method, apparatus and system

Abstract

The present application discloses a product out-of-stock detection method, apparatus and system. The method comprises: obtaining a display image of objects at each level of a shelf in a fully stocked state; determining the type of each level, and using a corresponding anchor box detection strategy to perform anchor box detection on the display image of each level in the fully stocked state; if the objects are individual products, on the basis of a plurality of historical display images, obtaining an empty-shelf feature vector set and at least one sample feature vector of the product corresponding to each anchor box; obtaining an image of a level to be analyzed, and then capturing an anchor box image of each anchor box therein; and for each anchor box, comparing a feature vector of the anchor box image of the anchor box with at least one sample feature vector of a product corresponding to the anchor box and the empty-shelf feature vector set, so as to determine a state of the anchor box; and for an anchor box in an out-of-stock state, on the basis of an extension area of an anchor box position of the anchor box, determining product information of an electronic price tag corresponding to the anchor box, and reporting the product information. The present application can achieve efficient and accurate shelf out-of-stock detection.

Description

Product Out-of-Stock Detection Methods, Devices and Systems

[0001] Related applications

[0002] This application claims priority to Chinese Patent Application No. 202510065108.7, filed on January 15, 2025, and incorporates the disclosure of the aforementioned patent application as part of this application. Technical Field

[0003] This application relates to the field of communication technology, and in particular to a method, apparatus and system for detecting out-of-stock items. Background Technology

[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0005] Merchants are increasingly reliant on efficient inventory management systems to ensure adequate product supply. However, traditional inventory management methods typically depend on manual inspection and record-keeping, which is not only time-consuming and labor-intensive but also prone to errors. This can lead to stockouts not being detected or replenished in a timely manner, thus impacting customer experience and sales performance. With the continuous development of the modern retail industry, automated, efficient, and accurate stockout detection methods have become a crucial requirement in the current retail landscape. Summary of the Invention

[0006] This application provides a method for detecting out-of-stock items, which can automatically, efficiently, and accurately detect out-of-stock items on shelves. The method includes:

[0007] Obtain the display images of all objects on each shelf level when they are fully stocked;

[0008] Based on the objects in each shelf level, determine the type of each level and use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level when it is full of goods, so as to obtain the anchor frame of each object in each level.

[0009] If the object is a single product, based on multiple historical display images of each shelf level, obtain a set of empty product feature vectors and at least one sample feature vector of the product corresponding to each anchor frame;

[0010] After obtaining the image of the layer to be analyzed, the corresponding anchor box detection strategy is adopted to obtain the anchor box of each object in the layer to be analyzed, and the anchor box image of each anchor box in the layer to be analyzed is extracted.

[0011] If the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and empty product feature vector set of the product corresponding to the anchor frame. Based on the comparison result, the state of the anchor frame is determined, and the state is out of stock or in stock.

[0012] For anchor frames that are out of stock, determine the corresponding electronic price tag based on the extended area of ​​the anchor frame's location.

[0013] Report the product information corresponding to the out-of-stock electronic price tags.

[0014] This application provides a product out-of-stock detection device, which can automatically, efficiently, and accurately detect out-of-stock items on shelves. The device includes:

[0015] The display image acquisition module is used to acquire display images of all objects on each shelf when they are fully stocked.

[0016] The anchor frame detection module is used to determine the type of each level based on the objects in each level of the shelf, and to use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level when it is full of goods, so as to obtain the anchor frame of each object in each level.

[0017] The feature vector acquisition module is used to obtain, if the object is a single product, a set of empty product feature vectors and at least one sample feature vector of the product corresponding to each anchor frame, based on multiple historical display images of each shelf level.

[0018] The anchor frame image cropping module is used to obtain the anchor frame of each object in the layer to be analyzed by adopting the corresponding anchor frame detection strategy after obtaining the image of the layer to be analyzed, and to crop the anchor frame image of each anchor frame in the layer to be analyzed.

[0019] Anchor frame status analysis module is used to compare the feature vector of the anchor frame image of each anchor frame with at least one sample feature vector and empty goods feature vector set of the product corresponding to the anchor frame if the object is a single product, and determine the status of the anchor frame based on the comparison result, wherein the status is out of stock or in stock.

[0020] The electronic price tag determination module is used to determine the corresponding electronic price tag for an out-of-stock anchor frame based on the extended area of ​​the anchor frame position.

[0021] The reporting module is used to report product information corresponding to out-of-stock electronic price tags.

[0022] This application provides a product shortage detection system that can automatically, efficiently, and accurately detect product shortages on shelves. The system includes a vision sensor and the aforementioned product shortage detection device.

[0023] The camera is used to capture images of all objects displayed on each shelf when it is fully stocked, as well as images of the shelf to be analyzed, and send them to the out-of-stock detection device.

[0024] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for detecting out-of-stock goods.

[0025] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described product shortage detection method.

[0026] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described out-of-stock detection method.

[0027] In this embodiment, display images of all objects on each shelf level in a fully stocked state are obtained; based on the objects in each shelf level, the type of each level is determined, and a corresponding anchor frame detection strategy is adopted to perform anchor frame detection on the display images of each level in a fully stocked state, obtaining the anchor frame of each object in each level; if the object is a single product, based on multiple historical display images of each shelf level, an empty product feature vector set and at least one sample feature vector of the product corresponding to each anchor frame are obtained; after obtaining the image of the level to be analyzed, the corresponding anchor frame detection strategy is adopted to obtain the anchor frame of each object in the level to be analyzed, and the anchor frame image of each anchor frame in the level to be analyzed is extracted; if the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector of the product corresponding to the anchor frame and the empty product feature vector set, and the state of the anchor frame is determined based on the comparison result, the state being out of stock or in stock; for an out-of-stock anchor frame, the electronic price tag corresponding to the anchor frame is determined based on the extended area of ​​the anchor frame position; the product information corresponding to the out-of-stock electronic price tag is reported. In the above process, the shelf display information of the full-stock state is restored by historical display information. According to the type of layer, the corresponding anchor frame detection strategy is determined, making the anchor frame detection more accurate. For a single product, the empty stock feature vector set and at least one sample feature vector of the product corresponding to each anchor frame can be obtained. Then, it is compared with the feature vector of the anchor frame image of each anchor frame in the layer to be analyzed to give an accurate out-of-stock judgment result. After determining the out-of-stock anchor frame position, the out-of-stock product information is found from one or more price tags in the area extending downward from the anchor frame. This process is highly efficient. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0029] Figure 1 is a flowchart of the product shortage detection method in an embodiment of this application;

[0030] Figure 2 is a schematic diagram of dividing each historical display image into multiple equally sized blocks in an embodiment of this application;

[0031] Figure 3 is a schematic diagram of the reference mask in an embodiment of this application;

[0032] Figure 4 is a schematic diagram of the mask image corresponding to the display image of all objects in the fully stocked state in the embodiment of this application;

[0033] Figure 5 is a schematic diagram of the first type of shelf hierarchy in the embodiments of this application;

[0034] Figure 6 is a schematic diagram of the second type of shelf hierarchy in the embodiments of this application;

[0035] Figure 7 is a schematic diagram of the third type of shelf hierarchy in the embodiments of this application;

[0036] Figure 8 is a schematic diagram of the shelf-level out-of-stock judgment logic of the first type in the embodiments of this application;

[0037] Figure 9 is a schematic diagram of the shelf-level out-of-stock judgment logic of the second type in the embodiments of this application;

[0038] Figure 10 is a schematic diagram of the shelf-level out-of-stock judgment logic of the third type in the embodiments of this application;

[0039] Figure 11 is a schematic diagram of the principle of finding electronic price tags by extending the anchor frame position of the anchor frame in an embodiment of this application;

[0040] Figure 12 is a schematic diagram of the principle of finding electronic price tags by extending the anchor frame position of the anchor frame in an embodiment of this application;

[0041] Figure 13 is a schematic diagram of the out-of-stock detection device in an embodiment of this application;

[0042] Figure 14 is a schematic diagram of the out-of-stock detection system in an embodiment of this application;

[0043] Figure 15 is a schematic diagram of the computer device in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0045] Figure 1 is a flowchart of the product shortage detection method in an embodiment of this application, including:

[0046] Step 101: Obtain the display images of all objects on each shelf level when they are fully stocked;

[0047] Step 102: Based on the objects in each shelf level, determine the type of each level and use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level in the full-stock state, so as to obtain the anchor frame of each object in each level.

[0048] Step 103: If the object is a single product, obtain the empty product feature vector set and at least one sample feature vector of the product corresponding to each anchor frame based on multiple historical display images of each shelf level.

[0049] Step 104: After obtaining the image of the layer to be analyzed, the corresponding anchor box detection strategy is adopted to obtain the anchor box of each object in the layer to be analyzed, and the anchor box image of each anchor box in the layer to be analyzed is extracted.

[0050] Step 105: If the object is a single product, for each anchor frame, compare the feature vector of the anchor frame image with at least one sample feature vector and empty product feature vector set of the product corresponding to the anchor frame. Based on the comparison result, determine the state of the anchor frame, which is either out of stock or in stock.

[0051] Step 106: For out-of-stock anchor frames, determine the corresponding electronic price tag based on the extended area of ​​the anchor frame position.

[0052] Step 107: Report the product information corresponding to the out-of-stock electronic price tags.

[0053] Compared with existing technologies, this application reconstructs the shelf display information of a fully stocked state through historical display information. Based on the type of layer, it determines the corresponding anchor frame detection strategy, making the anchor frame detection more accurate. For a single product, it can obtain the empty stock feature vector set and at least one sample feature vector of the product corresponding to each anchor frame. Then, it compares the feature vector with the feature vector of the anchor frame image of each anchor frame in the layer to be analyzed, and gives an accurate out-of-stock judgment result. After determining the out-of-stock anchor frame position, it searches for out-of-stock product information from one or more price tags in the area extending downward from the anchor frame. This process is highly efficient.

[0054] Each step is described in detail below.

[0055] In step 101, obtain the display images of all objects on each shelf level when they are fully stocked;

[0056] In this application embodiment, there are two methods for obtaining the display images of all objects in a fully stocked state.

[0057] In one embodiment, obtaining display images of all objects on each shelf level in a fully stocked state includes:

[0058] Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor;

[0059] Each historical display image is divided into multiple proportionally proportioned panels;

[0060] For multiple tiles in the same position in the historical display images, select the tile with the largest object detection area as the sub-tile to be stitched together;

[0061] Combine all the sub-images to be stitched together to form a display image of all objects at each level in a fully stocked state.

[0062] Specifically, the visual sensor can be a camera, which can be fixed to take pictures. For example, at a certain moment in the past, product A was out of stock while product B was not, and at another moment, product B was out of stock while product A was not. Based on these two historical display images, the state where products A and B were simultaneously fully stocked can be obtained. Similarly, the state when the shelves are completely full can be restored. For example, Figure 2 is a schematic diagram of dividing each historical display image into multiple equally sized blocks in an embodiment of this application. In Figure 2, it is divided into 3×3=9 blocks. For the position corresponding to the first block, the block with the largest object detection area is selected from multiple blocks as the sub-block to be stitched. After stitching, the display image of all objects at each level in the fully stocked state can be obtained.

[0063] In one embodiment, obtaining display images of all objects on each shelf level in a fully stocked state includes:

[0064] Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor;

[0065] The first historical display image is used as the reference image. The reference image is then used for detection image recognition to obtain multiple detection boxes.

[0066] Multiple detection boxes in the baseline image are combined to form a baseline mask image. Areas in the baseline mask image containing objects are marked with the first color, and areas without objects are marked with the second color.

[0067] The detection image recognition is performed sequentially on the historical display images except the reference image to obtain multiple candidate detection boxes. When the area of ​​a candidate detection box is greater than the area of ​​the first color-marked part in the reference mask image at the corresponding position by a preset ratio, the mask image of the candidate detection box is updated to the reference mask image at the corresponding position. The object box image detected by the candidate detection box is pasted on the reference image at the corresponding position. After the area of ​​the candidate detection boxes of all historical display images is determined, the display image of all objects of each shelf level in the full-stock state is obtained.

[0068] Figure 3 is a schematic diagram of the reference mask in an embodiment of this application. In Figure 3, areas with objects are marked in white, and areas without objects are marked in black. Of course, other colors can also be used for marking, and this application does not limit this. Figure 4 is a schematic diagram of the mask corresponding to the display image of all objects in a fully stocked state in an embodiment of this application. It can be seen that the mask corresponding to the display image of all objects in a fully stocked state is more complete than the original reference mask.

[0069] In step 102, based on the objects in each shelf level, the type of each level is determined, and the corresponding anchor frame detection strategy is adopted to perform anchor frame detection on the display image of each level in the full-stock state, so as to obtain the anchor frame of each object in each level.

[0070] In one embodiment, determining the type of each shelf level based on the objects in each level includes:

[0071] If the object in each shelf level is a single product, then the level is determined to be of the first type, and the anchor frame detection strategy is to place the single product in an anchor frame for anchor frame detection.

[0072] If the objects in each shelf level are bulk goods groups, the level is determined to be the second type, and the anchor frame detection strategy is to place all identical goods within a first distance into one anchor frame for anchor frame detection.

[0073] If the objects in each shelf level are boxed goods, the level is determined to be of type 3, and the anchor frame detection strategy is to place an identical group of boxed goods in one anchor frame for anchor frame detection.

[0074] In this embodiment, the hierarchy includes three cases, therefore, it is necessary to classify the shelf hierarchy when deploying cameras. Classification can be based on the objects whose hierarchy is obtained.

[0075] In addition, this application embodiment supports the classification being formed by manual configuration. That is, when deploying a camera, the classification of its corresponding shelf level can be manually configured to the camera in advance. The camera can automatically classify the images based on the content being captured. For example, by using the captured shelf level number, the corresponding goods category in the backend server can be found, and the shelf level category can be determined based on different goods categories.

[0076] Figure 5 is a schematic diagram of the first type of shelf hierarchy in this application embodiment. For this type of shelf, the display is relatively standard and the goods are placed in a standardized manner. This type of shelf ensures that there is only one product in each anchor frame, and the product has a clear display surface. Therefore, the anchor frame detection strategy is to place a single product in an anchor frame for anchor frame detection. The dashed rectangle is the anchor frame. Figure 6 is a schematic diagram of the second type of shelf hierarchy in this application embodiment. This type of shelf generally displays some bulk goods, such as yogurt, sausages, toothbrushes, etc. in cardboard boxes. In this case, it is not possible to guarantee that there is only a single product in the anchor frame, and the state of the goods in the box is complicated. The position where the customer picks up the goods may be random, and it is not possible to guarantee the generality of picking from the outside to the inside. The anchor frame detection strategy is to place all identical goods within a distance of less than a first distance in an anchor frame for anchor frame detection. The principle is that identical goods that are close to each other are placed in an anchor frame. If there are boxes, then one box is an anchor frame. In Figure 6, the dashed rectangle is the anchor frame, and the solid rectangle is the product detection frame. Figure 7 is a schematic diagram of the third type of shelf hierarchy in the embodiments of this application. This type of shelf is generally used to place boxed products, such as large bottles of water. The anchor frame detection strategy is to place a group of identical boxed products in an anchor frame for anchor frame detection. In Figure 7, the dashed rectangle is the anchor frame and the solid rectangle is the product detection frame.

[0077] In one embodiment, anchor frame detection is performed on the display images of each level in a fully stocked state to obtain the anchor frame for each object in each level, including:

[0078] Anchor frames are detected for the display images of each level under full stock status to obtain preliminary detected anchor frames;

[0079] The anchor frames from the initial inspection are subjected to quality inspection. Based on the quality inspection results, the anchor frames from the initial inspection are corrected to obtain the anchor frames for each object. The quality inspection includes checking the integrity of the object within the anchor frame, checking for the absence of images within the anchor frame, and checking for the presence of images without anchor frames.

[0080] Specifically, the anchor frames obtained according to the anchor frame detection strategy are the initial detection anchor frames. To improve their accuracy, quality inspection is required. The anchor frame object integrity check checks whether the object is completely contained within the anchor frame. If not, a complete anchor frame of the object is captured again from the display image. Anchor frames can also be manually modified through the operation interface, including adding, deleting, and changing regions. The anchor frame no image check checks whether there is no image within the anchor frame. If so, the anchor frame is deleted. The anchor frame no image check checks whether there are objects in the display image that have not generated anchor frames. If so, the anchor frame detection is re-performed for the objects in the display image.

[0081] In step 103, if the object is a single product, based on multiple historical display images of each shelf level, obtain the empty product feature vector set and at least one sample feature vector of the product corresponding to each anchor frame.

[0082] Specifically, this step targets a single product; feature vectors for other objects do not need to be collected. The multiple historical display images mentioned here are not display images of the previous fully stocked state, but rather a large number of historical display images of each level of the actual shelves, from which sample feature vectors and empty stock feature vector sets are obtained.

[0083] In one embodiment, based on multiple historical display images of each shelf level, a set of empty stock feature vectors and at least one sample feature vector of the product corresponding to each anchor frame are obtained, including:

[0084] Extract at least one sample image from each anchor frame from multiple historical display images of each shelf level;

[0085] For each sample image, if the sample image does not contain any goods, extract the feature vector of the sample image as the empty goods feature vector and add it to the empty goods feature vector set.

[0086] For each sample image, if the sample image contains a product, extract the feature vector of the sample image and use it as a sample feature vector of the product corresponding to the anchor box.

[0087] Specifically, since there are multiple historical display images, multiple sample images can be obtained for each anchor frame. If the sample image contains a product, the feature vector of that sample image is extracted and used as a sample feature vector of the product corresponding to the anchor frame. Thus, a single anchor frame often yields multiple sample feature vectors. All empty product feature vectors, regardless of which product's sample image they are obtained from, are added to the empty product feature vector set.

[0088] In one embodiment, after obtaining the empty cargo feature vector set and at least one sample feature vector of the commodity corresponding to each anchor frame, the method further includes:

[0089] Establish a mapping relationship between anchor frames and at least one sample feature vector, and store the mapping relationship, at least one sample feature vector, and empty cargo feature vector set into the feature vector library. The mapping relationship includes anchor frame identifier, anchor frame position, and at least one sample feature vector.

[0090] Specifically, when comparing sample feature vectors later, they are read from the feature vector library, and each anchor box is marked with an anchor box identifier for easy querying.

[0091] In step 104, after obtaining the image of the layer to be analyzed, the corresponding anchor frame detection strategy is adopted to obtain the anchor frame of each object in the layer to be analyzed, and the anchor frame image of each anchor frame in the layer to be analyzed is extracted.

[0092] Specifically, each anchor frame position in the analyzed layer also has a corresponding anchor frame identifier, and its anchor frame image can be obtained. Because the difficulty of identifying and detecting goods on different shelves varies, different out-of-stock judgment logic needs to be adopted for different shelf layers.

[0093] In step 105, if the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and empty product feature vector set of the product corresponding to the anchor frame. Based on the comparison result, the state of the anchor frame is determined, and the state is out of stock or in stock.

[0094] Since the difficulty of identifying and detecting goods on different shelves varies, different out-of-stock judgment logics need to be adopted for different shelves. This application embodiment targets three types of levels. Step 105 targets a single product, i.e., the first type of level.

[0095] Figure 8 is a schematic diagram of the shelf-level out-of-stock judgment logic of the first type in an embodiment of this application. In one embodiment, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and an empty stock feature vector set of the corresponding product. Based on the comparison result, the state of the anchor frame is determined, including:

[0096] For each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with the feature vector of at least one sample of the product corresponding to the anchor frame for the first time.

[0097] When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, the status of the anchor frame is determined to be "in stock".

[0098] Otherwise, perform a second similarity comparison between the feature vector of the anchor frame image and each empty cargo feature vector in the empty cargo feature vector set;

[0099] When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, the state of the anchor frame is determined to be out of stock;

[0100] Otherwise, perform product detection on the anchor frame image of the anchor frame to obtain the product detection box;

[0101] If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the status of the anchor frame is determined to be in stock; otherwise, it is out of stock.

[0102] Specifically, when performing the first similarity comparison between the feature vector of the anchor frame image and at least one sample feature vector of the product corresponding to the anchor frame, the feature vector of the anchor frame image is compared with the sample feature vectors in turn, the similarity between the vectors is calculated, and the maximum similarity is taken. If the maximum similarity is greater than the first similarity threshold, the state of the anchor frame is determined to be in stock; otherwise, the next step of judgment is performed.

[0103] The next step is to compare empty cargo with cargo. The set of empty cargo feature vectors can be represented as F. empty ={F empty-1 ,F empty-2 ,F empty-3 ,F empty-4 The anchor frame contains multiple empty cargo feature vectors. The feature vectors of the anchor frame image are compared with the empty cargo feature vectors for the second time. That is, the similarity between the vectors is calculated and the maximum similarity is taken. If the maximum similarity is greater than the second similarity threshold, the state of the anchor frame is determined to be out of stock. Otherwise, the next step is to perform the product detection.

[0104] By following the steps above, the probability of product inspection can be greatly reduced. By comparing the first and second similarity, the conclusion of whether the product is out of stock can be obtained with a high probability, thereby greatly improving the efficiency of out-of-stock judgment.

[0105] Figure 9 is a schematic diagram of the shelf-level out-of-stock judgment logic of the second type in an embodiment of this application. In one embodiment, the method further includes:

[0106] If the object is a bulk goods group, perform goods detection on the anchor frame image of each anchor frame in the analysis level to obtain multiple goods detection boxes;

[0107] For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to that anchor frame exceeds the area threshold, the status of that anchor frame is determined to be "in stock".

[0108] The above embodiments are for the second type of shelf level. The out-of-stock judgment strategy for this type of shelf is as follows: only the product detection frame of the product in the anchor frame is judged, the total area S1 of all product detection frames in the anchor frame is taken and compared with the current anchor frame area S2. If the detection result S1 / S2 exceeds the area threshold, it is considered that the current goods status does not need to be replenished; otherwise, an out-of-stock report is required.

[0109] Figure 10 is a schematic diagram of the shelf-level out-of-stock judgment logic of the third type in the embodiments of this application. In one embodiment, the method further includes:

[0110] If the object is a boxed product, determine the lower edge of each anchor frame in the level to be analyzed as the baseline reference line of that anchor frame;

[0111] Perform product detection on the anchor frame image of each anchor frame in the analysis level to obtain product detection results;

[0112] If the product detection result is no product detection frame, the status of that anchor frame is determined to be out of stock;

[0113] If the product detection result shows that there is a product detection box, the product check box that is closest to the baseline reference line in the product detection box is determined as the nearest product detection box;

[0114] If the distance between the most recent product detection frame and the baseline reference line exceeds the distance threshold, the status of the anchor frame is determined to be out of stock; otherwise, the status of the anchor frame is determined to be in stock.

[0115] The above embodiment targets the third type of shelf level. The lower edge of the anchor frame is defined as the baseline reference line for the current anchor frame. Within the anchor frame, only the distance between the bottommost product detection box and the anchor frame baseline is checked. If the distance exceeds a certain threshold, the outermost layer of products is considered empty, and a stockout report is required. Another scenario is that no products are detected in the current anchor frame; in this case, a stockout status is reported directly.

[0116] In step 106, for out-of-stock anchor frames, the electronic price tag corresponding to the anchor frame is determined based on the extended area of ​​the anchor frame position.

[0117] In one embodiment, for an out-of-stock anchor frame, determining the corresponding electronic price tag based on the extended area of ​​the anchor frame's location includes:

[0118] For anchor frames that are out of stock, the extension area is determined based on the location of the anchor frame.

[0119] If there is at least one electronic price tag in the extended area, use at least one electronic price tag as the electronic price tag corresponding to the anchor frame;

[0120] If there is no electronic price tag in the extended area, the nearest electronic price tag on both sides of the extended area will be used as the electronic price tag corresponding to the anchor frame.

[0121] Specifically, the binding information between the location of the electronic shelf tag and product information (e.g., Stock Keeping Unit, SKU) is stored in the backend server. Once a stock-out anchor frame (including its identifier and location) is identified, the corresponding electronic shelf tag is located by extending downwards from the anchor frame location. Based on the product information bound to the electronic shelf tag, the backend server retrieves the product information corresponding to the stock-out anchor frame and reports it to the merchant. This allows the merchant to retrieve the specific product and replenish the stock at the target location. It should be noted that the downward extension area may cover multiple electronic shelf tags. Figure 11 illustrates the principle of finding electronic shelf tags by extending the anchor frame location in this embodiment. In this case, depending on the actual situation, the top three or top two possible product information results can be determined for reporting, and the merchant can replenish the product.

[0122] It's also possible that the downward extension area of ​​the anchor frame may not cover the price tag. Figure 12 shows the second principle diagram of finding electronic price tags through the extension area of ​​the anchor frame position in this embodiment. In this case, it is necessary to search from the downward extension area to the left and right for at least one electronic price tag closest to the current extension area, and report at least one product information that may be out of stock. If it is necessary to report a unique out-of-stock SKU, it is also possible to compare the center position of the candidate out-of-stock price tag with the center position of the out-of-stock anchor frame, calculate the absolute value of the difference between the x-coordinate of the price tag center and the x-coordinate of the anchor frame center, select the product information (e.g., SKU) corresponding to the price tag with the smallest absolute value, and report the out-of-stock product information.

[0123] In one embodiment, the method further includes:

[0124] The visual server captures an image of the flashing electronic price tag. The backend server sends a flashing command to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing command.

[0125] Detect the price tag image and obtain the position of the price tag detection box;

[0126] Update the position of the electronic price tag based on the position of the price tag detection box.

[0127] Specifically, the method of determining the out-of-stock anchor frame and reporting out-of-stock items relies on the accuracy of the price tag coordinates. In real-world scenarios, situations such as product re-binding, new product binding, and price tag movement occur every day, so developing a price tag position update strategy is crucial.

[0128] After determining the initial price tag coordinates, the latest price tag binding relationships are retrieved from the price tag backend server at a certain frequency each day. If a price tag binding relationship changes, or a new price tag is bound, the positions of these changed price tags need to be tracked. The backend server issues a flash command to the price tags whose binding relationships have changed. After issuing the flash command, the camera takes pictures of all electronic price tags in the coverage area, obtaining images of all electronic price tags. Then, the images of the electronic price tags are detected to obtain the price tag detection box positions D = {(x1, y1, w1, h1), (x2, y2, w3, h4), ...}, where x i y i Here, wi and hi represent the pixel coordinates of the center point of the price tag detection frame, and wi and hi represent the pixel width and height of the price tag detection frame, respectively. After the flash command is issued, the target electronic price tag begins to flash. The camera captures the position of the price tag detection frame at the flash location, and the position of the price tag for which the flash command was sent is determined based on the price tag detection frame position, thereby updating the price tag position in the database. The latest position of the price tag is maintained at a fixed frequency in this way to ensure that the out-of-stock anchor frame can be correctly identified with the out-of-stock product information.

[0129] In step 107, report the product information corresponding to the out-of-stock electronic price tags. Specifically, this can be reported to the backend server for processing by administrators. In addition to reporting out-of-stock product information, the out-of-stock anchor frames, i.e., anchor frame identifiers and anchor frame positions, can also be reported so that merchants can replenish stock at the accurate locations.

[0130] This application also proposes a product shortage detection device, the principle of which is similar to the product shortage detection method, and will not be described in detail here.

[0131] Figure 13 is a schematic diagram of the out-of-stock detection device in an embodiment of this application, including:

[0132] The display image acquisition module 1301 is used to acquire display images of all objects on each shelf in a fully stocked state.

[0133] Anchor frame detection module 1302 is used to determine the type of each level based on the objects in each level of the shelf, and to use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level in the full-stock state, so as to obtain the anchor frame of each object in each level.

[0134] The feature vector acquisition module 1303 is used to obtain, if the object is a single product, a set of empty product feature vectors and at least one sample feature vector of the product corresponding to each anchor frame, based on multiple historical display images of each shelf level.

[0135] The anchor frame image cropping module 1304 is used to obtain the anchor frame of each object in the layer to be analyzed by adopting the corresponding anchor frame detection strategy after obtaining the image of the layer to be analyzed, and to crop the anchor frame image of each anchor frame in the layer to be analyzed.

[0136] Anchor frame status analysis module 1305 is used to compare the feature vector of the anchor frame image of each anchor frame with at least one sample feature vector and empty goods feature vector set of the product corresponding to the anchor frame if the object is a single product, and determine the status of the anchor frame based on the comparison result, which is either out of stock or in stock.

[0137] The electronic price tag determination module 1306 is used to determine the electronic price tag corresponding to the anchor frame for the out-of-stock anchor frame based on the extended area of ​​the anchor frame position.

[0138] The reporting module 1307 is used to report product information corresponding to out-of-stock electronic price tags.

[0139] In one embodiment, the display image acquisition module is used to:

[0140] Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor;

[0141] Each historical display image is divided into multiple proportionally proportioned panels;

[0142] For multiple tiles in the same position in the historical display images, select the tile with the largest object detection area as the sub-tile to be stitched together;

[0143] Combine all the sub-images to be stitched together to form a display image of all objects at each level in a fully stocked state.

[0144] In one embodiment, the display image acquisition module is used to:

[0145] Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor;

[0146] The first historical display image is used as the reference image. The reference image is then used for detection image recognition to obtain multiple detection boxes.

[0147] Multiple detection boxes in the baseline image are combined to form a baseline mask image. Areas in the baseline mask image containing objects are marked with the first color, and areas without objects are marked with the second color.

[0148] The detection image recognition is performed sequentially on the historical display images except the reference image to obtain multiple candidate detection boxes. When the area of ​​a candidate detection box is greater than the area of ​​the first color-marked part in the reference mask image at the corresponding position by a preset ratio, the mask image of the candidate detection box is updated to the reference mask image at the corresponding position. The object box image detected by the candidate detection box is pasted on the reference image at the corresponding position. After the area of ​​the candidate detection boxes of all historical display images is determined, the display image of all objects of each shelf level in the full-stock state is obtained.

[0149] In one embodiment, the anchor frame detection module is used for:

[0150] If the object in each shelf level is a single product, then the level is determined to be of the first type, and the anchor frame detection strategy is to place the single product in an anchor frame for anchor frame detection.

[0151] If the objects in each shelf level are bulk goods groups, the level is determined to be the second type, and the anchor frame detection strategy is to place all identical goods within a first distance into one anchor frame for anchor frame detection.

[0152] If the objects in each shelf level are boxed goods, the level is determined to be of type 3, and the anchor frame detection strategy is to place an identical group of boxed goods in one anchor frame for anchor frame detection.

[0153] In one embodiment, the anchor frame detection module is used for:

[0154] Anchor frames are detected for the display images of each level under full stock status to obtain preliminary detected anchor frames;

[0155] The anchor frames from the initial inspection are subjected to quality inspection. Based on the quality inspection results, the anchor frames from the initial inspection are corrected to obtain the anchor frames for each object. The quality inspection includes checking the integrity of the object within the anchor frame, checking for the absence of images within the anchor frame, and checking for the presence of images without anchor frames.

[0156] In one embodiment, the feature vector acquisition module is used to:

[0157] Extract at least one sample image from each anchor frame from multiple historical display images of each shelf level;

[0158] For each sample image, if the sample image does not contain any goods, extract the feature vector of the sample image as the empty goods feature vector and add it to the empty goods feature vector set.

[0159] For each sample image, if the sample image contains a product, extract the feature vector of the sample image and use it as a sample feature vector of the product corresponding to the anchor box.

[0160] In one embodiment, the feature vector acquisition module is further configured to:

[0161] After obtaining the empty cargo feature vector set and at least one sample feature vector of the commodity corresponding to each anchor box, the process also includes:

[0162] Establish a mapping relationship between anchor frames and at least one sample feature vector, and store the mapping relationship, at least one sample feature vector, and empty cargo feature vector set into the feature vector library. The mapping relationship includes anchor frame identifier, anchor frame position, and at least one sample feature vector.

[0163] In one embodiment, the anchor frame state analysis module is used for:

[0164] For each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with the feature vector of at least one sample of the product corresponding to the anchor frame for the first time.

[0165] When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, the status of the anchor frame is determined to be "in stock".

[0166] Otherwise, perform a second similarity comparison between the feature vector of the anchor frame image and each empty cargo feature vector in the empty cargo feature vector set;

[0167] When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, the state of the anchor frame is determined to be out of stock;

[0168] Otherwise, perform product detection on the anchor frame image of the anchor frame to obtain the product detection box;

[0169] If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the status of the anchor frame is determined to be in stock; otherwise, it is out of stock.

[0170] In one embodiment, the anchor frame state analysis module is used for:

[0171] If the object is a bulk goods group, perform goods detection on the anchor frame image of each anchor frame in the analysis level to obtain multiple goods detection boxes;

[0172] For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to that anchor frame exceeds the area threshold, the status of that anchor frame is determined to be "in stock".

[0173] In one embodiment, the anchor frame state analysis module is used for:

[0174] If the object is a boxed product, determine the lower edge of each anchor frame in the level to be analyzed as the baseline reference line of that anchor frame;

[0175] Perform product detection on the anchor frame image of each anchor frame in the analysis level to obtain product detection results;

[0176] If the product detection result is no product detection frame, the status of that anchor frame is determined to be out of stock;

[0177] If the product detection result shows that there is a product detection box, the product check box that is closest to the baseline reference line in the product detection box is determined as the nearest product detection box;

[0178] If the distance between the most recent product detection frame and the baseline reference line exceeds the distance threshold, the status of the anchor frame is determined to be out of stock; otherwise, the status of the anchor frame is determined to be in stock.

[0179] In one embodiment, the electronic price tag determination module is used for:

[0180] For anchor frames that are out of stock, the extension area is determined based on the location of the anchor frame.

[0181] If there is at least one electronic price tag in the extended area, use at least one electronic price tag as the electronic price tag corresponding to the anchor frame;

[0182] If there is no electronic price tag in the extended area, the nearest electronic price tag on both sides of the extended area will be used as the electronic price tag corresponding to the anchor frame.

[0183] In one embodiment, the device further includes a price tag position updating module, used for:

[0184] The visual server captures an image of the flashing electronic price tag. The backend server sends a flashing command to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing command.

[0185] Detect the price tag image and obtain the position of the price tag detection box;

[0186] Update the position of the electronic price tag based on the position of the price tag detection box.

[0187] This application also proposes a product shortage detection system. Figure 14 is a schematic diagram of the product shortage detection system in this application. The system includes: a visual sensor 1401 and the aforementioned product shortage detection device 1402.

[0188] The vision sensor 1401 is used to capture images of all objects displayed on each shelf when it is fully stocked, as well as images of the shelf to be analyzed, and send them to the out-of-stock detection device.

[0189] The visual sensor can be a camera or other device capable of taking pictures. The visual sensor also captures flashing price tags. The back-end management system includes a feature vector library for storing mapping relationships and at least one set of sample feature vectors and empty cargo feature vectors.

[0190] The out-of-stock detection system proposed in this application interacts with the back-end management system. The visual sensor can capture images of flashing price tags and store the captured images in the back-end management system. The captured price tag positions are uploaded to the back-end management module for unified updates of price tag positions.

[0191] The main functions of the backend management system are: to retrieve the latest price tag product binding relationships, obtain the changed price tag representations, and issue flashing commands to specified price tags; to configure and modify the anchor frame positions; to visually display the out-of-stock judgment results; to receive price tag position information from the camera module and maintain the latest price tag position relationships, etc.

[0192] In summary, the methods, apparatus, and systems proposed in this application have the following beneficial effects:

[0193] Obtain display images of all objects on each shelf level in a fully stocked state; determine the type of each shelf level based on the objects in each shelf level, and use the corresponding anchor frame detection strategy to perform anchor frame detection on the display images of each shelf level in a fully stocked state, obtaining the anchor frame of each object in each shelf level; if the object is a single product, obtain the empty product feature vector set and at least one sample feature vector of the product corresponding to each anchor frame based on multiple historical display images of each shelf level; after obtaining the image of the shelf level to be analyzed, use the corresponding anchor frame detection strategy to obtain the anchor frame of each object in the shelf level to be analyzed, and extract the anchor frame image of each anchor frame in the shelf level to be analyzed; if the object is a single product, for each anchor frame, compare the feature vector of the anchor frame image of the anchor frame with at least one sample feature vector of the product corresponding to the anchor frame and the empty product feature vector set, and determine the state of the anchor frame based on the comparison result, which is either out of stock or in stock; for out-of-stock anchor frames, determine the electronic price tag corresponding to the anchor frame based on the extended area of ​​the anchor frame position; report the product information corresponding to the out-of-stock electronic price tag. Through the above process, the shelf display information of the full-stock state is restored by using historical display information. According to the type of layer, the corresponding anchor frame detection strategy can be determined, making the anchor frame detection more accurate. For a single product, the empty stock feature vector set and at least one sample feature vector of the product corresponding to each anchor frame can be obtained. Then, it is compared with the feature vector of the anchor frame image of each anchor frame in the layer to be analyzed to give an accurate out-of-stock judgment result. After determining the location of the out-of-stock anchor frame, the out-of-stock product information is found from one or more price tags in the area extending downward from the anchor frame. This process is highly efficient.

[0194] This application embodiment also provides a computer device. Figure 15 is a schematic diagram of the computer device in this application embodiment. The computer device 1500 includes a memory 1510, a processor 1520, and a computer program 1530 stored in the memory 1510 and executable on the processor 1520. When the processor 1520 executes the computer program 1530, it implements the above-mentioned product shortage detection method.

[0195] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described product shortage detection method.

[0196] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described out-of-stock detection method.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0201] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting out-of-stock items, characterized in that, include: Obtain the display images of all objects on each shelf level when they are fully stocked; Based on the objects in each shelf level, determine the type of each level and use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level when it is full of goods, so as to obtain the anchor frame of each object in each level. If the object is a single product, based on multiple historical display images of each shelf level, obtain a set of empty product feature vectors and at least one sample feature vector of the product corresponding to each anchor frame; After obtaining the image of the layer to be analyzed, the corresponding anchor box detection strategy is adopted to obtain the anchor box of each object in the layer to be analyzed, and the anchor box image of each anchor box in the layer to be analyzed is extracted. If the object is a single product, for each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with at least one sample feature vector and empty product feature vector set of the product corresponding to the anchor frame. Based on the comparison result, the state of the anchor frame is determined, and the state is out of stock or in stock. For anchor frames that are out of stock, determine the corresponding electronic price tag based on the extended area of ​​the anchor frame's location. Report the product information corresponding to the out-of-stock electronic price tags.

2. The method as described in claim 1, characterized in that, Obtain display images of all objects on each shelf level when fully stocked, including: Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor; Each historical display image is divided into multiple proportionally proportioned panels; For multiple tiles in the same position in the historical display images, select the tile with the largest object detection area as the sub-tile to be stitched together; Combine all the sub-images to be stitched together to form a display image of all objects at each level in a fully stocked state.

3. The method as described in claim 1, characterized in that, Obtain display images of all objects on each shelf level when fully stocked, including: Obtain multiple historical display images of equal size for each level of the shelf, captured by a visual sensor; The first historical display image is used as the reference image. The reference image is then used for detection image recognition to obtain multiple detection boxes. Multiple detection boxes in the baseline image are used to form a baseline mask image. In the baseline mask image, the areas with objects are marked with a first color, and the areas without objects are marked with a second color. The detection image recognition is performed sequentially on the historical display images except the reference image to obtain multiple candidate detection boxes. When the area of ​​a candidate detection box is greater than the area of ​​the first color-marked part in the reference mask image at the corresponding position by a preset ratio, the mask image of the candidate detection box is updated to the reference mask image at the corresponding position. The object box image detected by the candidate detection box is pasted on the reference image at the corresponding position. After the area of ​​the candidate detection boxes of all historical display images is determined, the display image of all objects of each shelf level in the full-stock state is obtained.

4. The method as described in claim 1, characterized in that, Based on the objects in each shelf level, determine the type of each level, including: If the object in each shelf level is a single product, then the level is determined to be of the first type, and the anchor frame detection strategy is to place the single product in an anchor frame for anchor frame detection. If the objects in each shelf level are bulk goods groups, the level is determined to be the second type, and the anchor frame detection strategy is to place all identical goods within a first distance into one anchor frame for anchor frame detection. If the objects in each shelf level are boxed goods, the level is determined to be of type 3, and the anchor frame detection strategy is to place an identical group of boxed goods in one anchor frame for anchor frame detection.

5. The method as described in claim 1, characterized in that, Anchor frame detection is performed on the display images of each level when the product is fully stocked to obtain the anchor frame for each object in each level, including: Anchor frames are detected for the display images of each level under full stock status to obtain preliminary detected anchor frames; The anchor frames that have undergone preliminary inspection are subjected to quality inspection. Based on the quality inspection results, the anchor frames that have undergone preliminary inspection are corrected to obtain the anchor frames for each object. The quality inspection includes checking the integrity of the object within the anchor frame, checking for the absence of an image within the anchor frame, and checking for the presence of an image without an anchor frame.

6. The method as described in claim 1, characterized in that, Based on multiple historical display images of each shelf level, obtain a set of empty stock feature vectors and at least one sample feature vector for the product corresponding to each anchor frame, including: Extract at least one sample image from each anchor frame from multiple historical display images of each shelf level; For each sample image, if the sample image does not contain any goods, extract the feature vector of the sample image as the empty goods feature vector and add it to the empty goods feature vector set. For each sample image, if the sample image contains a product, extract the feature vector of the sample image and use it as a sample feature vector of the product corresponding to the anchor box.

7. The method as described in claim 1, characterized in that, After obtaining the empty cargo feature vector set and at least one sample feature vector of the commodity corresponding to each anchor box, the process also includes: Establish a mapping relationship between anchor frames and at least one sample feature vector, and store the mapping relationship, at least one sample feature vector, and empty cargo feature vector set into a feature vector library. The mapping relationship includes anchor frame identifier, anchor frame position, and at least one sample feature vector.

8. The method as described in claim 1, characterized in that, For each anchor frame, the feature vector of the anchor frame image is compared with at least one sample feature vector and an empty cargo feature vector set of the corresponding product. Based on the comparison result, the state of the anchor frame is determined, including: For each anchor frame, the feature vector of the anchor frame image of the anchor frame is compared with the feature vector of at least one sample of the product corresponding to the anchor frame for the first time. When the maximum similarity of the first similarity comparison result is greater than the first similarity threshold, the status of the anchor frame is determined to be "in stock". Otherwise, perform a second similarity comparison between the feature vector of the anchor frame image and each empty cargo feature vector in the empty cargo feature vector set; When the maximum similarity of the second similarity comparison result is greater than the second similarity threshold, the state of the anchor frame is determined to be out of stock; Otherwise, perform product detection on the anchor frame image of the anchor frame to obtain the product detection box; If the intersection area between the anchor frame and the product detection frame exceeds the intersection threshold, the status of the anchor frame is determined to be in stock; otherwise, it is out of stock.

9. The method as described in claim 1, characterized in that, Also includes: If the object is a bulk commodity group, perform commodity detection on the anchor frame image of each anchor frame in the analysis level to obtain multiple commodity detection boxes; For each anchor frame, if the ratio of the total area of ​​multiple product detection frames to that anchor frame exceeds the area threshold, the status of that anchor frame is determined to be "in stock".

10. The method as described in claim 1, characterized in that, Also includes: If the object is a boxed product, the lower edge of each anchor frame in the level to be analyzed is determined as the reference line of that anchor frame; Perform product detection on the anchor frame image of each anchor frame in the analysis level to obtain product detection results; If the product detection result is no product detection frame, the status of that anchor frame is determined to be out of stock; If the product detection result shows that there is a product detection box, the product check box that is closest to the baseline reference line in the product detection box is determined as the nearest product detection box; If the distance between the most recent product detection frame and the baseline reference line exceeds the distance threshold, the status of the anchor frame is determined to be out of stock; otherwise, the status of the anchor frame is determined to be in stock.

11. The method as described in claim 1, characterized in that, For out-of-stock anchor frames, determine the corresponding electronic price tag based on the extended area of ​​the anchor frame's location, including: For anchor frames that are out of stock, the extension area is determined based on the location of the anchor frame. If there is at least one electronic price tag in the extended area, the at least one electronic price tag shall be used as the electronic price tag corresponding to the anchor frame; If there is no electronic price tag in the extended area, the nearest electronic price tag on both sides of the extended area will be used as the electronic price tag corresponding to the anchor frame.

12. The method as described in claim 1, characterized in that, Also includes: The visual server captures an image of the flashing electronic price tag. The backend server sends a flashing command to the electronic price tag when the binding relationship between the electronic price tag and the product information changes, so that the electronic price tag starts to flash after receiving the flashing command. The price tag image is detected to obtain the position of the price tag detection box; Update the position of the electronic price tag based on the position of the price tag detection box.

13. A product shortage detection device, characterized in that, include: The display image acquisition module is used to acquire display images of all objects on each shelf when they are fully stocked. The anchor frame detection module is used to determine the type of each level based on the objects in each level of the shelf, and to use the corresponding anchor frame detection strategy to perform anchor frame detection on the display image of each level when it is full of goods, so as to obtain the anchor frame of each object in each level. The feature vector acquisition module is used to obtain, if the object is a single product, a set of empty product feature vectors and at least one sample feature vector of the product corresponding to each anchor frame, based on multiple historical display images of each shelf level. The anchor frame image cropping module is used to obtain the anchor frame of each object in the layer to be analyzed by adopting the corresponding anchor frame detection strategy after obtaining the image of the layer to be analyzed, and to crop the anchor frame image of each anchor frame in the layer to be analyzed. Anchor frame status analysis module is used to compare the feature vector of the anchor frame image of each anchor frame with at least one sample feature vector and empty goods feature vector set of the product corresponding to the anchor frame if the object is a single product, and determine the status of the anchor frame based on the comparison result, wherein the status is out of stock or in stock. The electronic price tag determination module is used to determine the corresponding electronic price tag for an out-of-stock anchor frame based on the extended area of ​​the anchor frame position. The reporting module is used to report product information corresponding to out-of-stock electronic price tags.

14. A product out-of-stock detection system, characterized in that, include: The visual sensor and the out-of-stock detection device as described in claim 13; The visual sensor is used to capture images of all objects displayed on each shelf level when they are fully stocked, as well as images of the shelf level to be analyzed, and send them to the out-of-stock detection device.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images