Shopping cart-based shelf scene perception method, apparatus and system
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HANSHOW TECH CO LTD
- Filing Date
- 2026-01-07
- Publication Date
- 2026-07-23
Smart Images
Figure CN2026071022_23072026_PF_FP_ABST
Abstract
Description
Shopping cart-based shelf scene perception methods, devices, and systems
[0001] Related applications
[0002] This application claims priority to Chinese Patent Application No. 202510079908.4, filed on January 17, 2025, and incorporates the disclosure of the aforementioned patent application as part of this application. Technical Field
[0003] This application relates to the field of communication technology, and in particular to a method, device and system for perceiving shelf scenes based on a shopping cart. Background Technology
[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.
[0005] In recent years, with the continuous development of technologies such as the Internet of Things, artificial intelligence, big data analysis, mobile payment and smart hardware, the supermarket industry is also constantly transforming towards intelligence and digitalization in order to better meet customer needs, improve customer shopping experience and optimize merchant operations. Shelf environment perception has become a new requirement. Conventional shelf environment perception requires the deployment of a large number of sensors, visual sensors, etc., which consumes a lot of resources and manpower.
[0006] With the continuous development of supermarkets and shopping malls, smart shopping carts have emerged in response to this trend. The main purpose of smart shopping carts is to simplify the traditional steps of shopping and payment, eliminating the need for customers to interact with cashiers at the checkout counter and improving the shopping experience. At the same time, for merchants, it can reduce the number of cashiers required, thereby reducing operating costs.
[0007] Therefore, applying smart shopping carts to shelf environment sensing can fully utilize smart shopping carts, reduce resource and manpower input, further optimize merchant operations, and improve the customer shopping experience. Currently, however, there is a lack of solutions for shelf scene sensing based on shopping carts. Summary of the Invention
[0008] This application provides a shelf scene perception method based on a shopping cart. Applied to a shopping cart, it can update the shelf's product layout diagram and out-of-stock information in real time based on the shopping cart, exhibiting strong shelf perception capabilities. The method includes:
[0009] As the shopping cart is pushed by the shopper, images of each shelf's product display are acquired via visual sensors on the shopping cart.
[0010] Identify the shelf scene corresponding to the image; when the shelf scene is an analyzable shelf layout, identify the product information of each product in the image.
[0011] Obtain a product display layout drawing, which is obtained after identifying the products on each shelf;
[0012] When the number of identified products exceeds a preset threshold, the location information of the shopping cart is diffused to obtain a point range that matches a predefined shelf location. Combining the identified product information and identification confidence level with the point range, the area to be updated and the product to be updated in the product display drawing are located. The point range includes at least two points.
[0013] The product to be updated in the image is cropped out from its current area to obtain a small image of the product to be updated.
[0014] The small image of the product to be updated is compared with the pre-stored image of the product to be updated. If the similarity is greater than the similarity threshold, the product to be updated is determined to be correct. The update status of the area to be updated is recorded according to the product to be updated. The update status record is used to update the update area of the product layout drawing.
[0015] This application provides a shelf scene sensing device based on a shopping cart. Applied to a shopping cart, it can update the shelf's product layout diagram and out-of-stock information in real time based on the shopping cart, exhibiting strong shelf sensing capabilities. The device includes:
[0016] The image acquisition module is used to acquire images of each shelf display of goods captured by the visual sensors on the shopping cart as the shopping cart is pushed by the shopper;
[0017] The shelf scene recognition module is used to identify the shelf scene corresponding to the image, and when the shelf scene is an analyzable shelf layout, it identifies the product information of each product in the image;
[0018] The product display layout drawing update module is used to obtain product display layout drawings, which are obtained after identifying the products on each shelf. When the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred to obtain a point interval matching a predefined shelf location. Combining the identified product information and identification confidence level with the point interval, the module locates the area to be updated and the product to be updated in the product display layout drawing. The point interval includes at least two points. The product to be updated is cropped from its location in the image to obtain a small image of the product to be updated. The small image of the product to be updated is compared with a pre-stored image of the product to be updated. If the similarity is greater than a similarity threshold, the product to be updated is determined to be correct. The update status of the area to be updated is recorded according to the product to be updated. The update status record is used to update the area to be updated in the product display layout drawing.
[0019] This application provides a shelf scene perception system based on a shopping cart, which can update the product layout diagram and out-of-stock information of the shelf in real time based on the shopping cart. The system has strong shelf perception capabilities. The system includes: a shelf scene perception server and a shopping cart corresponding to the aforementioned device.
[0020] An environment-aware server is used for:
[0021] Receive and store the product layout diagram sent from the shopping cart.
[0022] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described shopping cart-based shelf scene perception method.
[0023] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described shopping cart-based shelf scene perception method.
[0024] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described shopping cart-based shelf scene perception method.
[0025] In this embodiment, during the process of the shopping cart being pushed by the shopper, images of the product displays on each shelf are acquired through visual sensors on the shopping cart; the shelf scene corresponding to the image is identified, and when the shelf scene is an analyzable shelf display, the product information of each product in the image is identified; when the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred to obtain a point interval matching a predefined shelf location, and the area to be updated and the product to be updated in the product display drawing are located by combining the identified product information and identification confidence level with the point interval, wherein the point interval includes at least two points; the product to be updated is cropped from the area in the image to obtain a small image of the product to be updated; the small image of the product to be updated is compared with a pre-stored image of the product to be updated, and when the similarity is greater than a similarity threshold, the product to be updated is determined to be correct, and the area to be updated is recorded as to be updated according to the product to be updated, and the record of the status to be updated is used to update the area to be updated in the product display drawing. Compared with the prior art, this application can identify the shelf scene corresponding to the image captured by the shopping cart, and then identify the product information in the shelf scene. In order to improve the accuracy and efficiency of updating the product display drawing, when the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred and diffused to obtain the location range. Then, combined with the product information, the area to be updated and the product to be updated are obtained. Then, only the area to be updated is updated according to the product to be updated. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0027] Figure 1 is a flowchart of the shelf scene perception method based on shopping cart in an embodiment of this application;
[0028] Figure 2 is a schematic diagram of the structure of the shelf scene perception device based on a shopping cart in an embodiment of this application;
[0029] Figure 3 is another structural schematic diagram of the shelf scene perception device based on a shopping cart in this embodiment of the application;
[0030] Figure 4 is a schematic diagram of a shelf scene perception system based on a shopping cart in an embodiment of this application;
[0031] Figure 5 is a schematic diagram of the computer device in an embodiment of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.
[0033] The smart shopping cart boasts a range of advanced features, including automatic scanning and checkout, navigation, item recognition, and weighing. Among these, self-service scanning and checkout is its core function. It utilizes smart hardware and algorithms to intelligently detect and identify products or barcodes, instantly updating the shopping list on a tablet, and finally, the shopper checks the list and proceeds to checkout.
[0034] However, in reality, the shopping cart is situated within a dazzling array of supermarket scenes. In addition to self-service shopping and self-checkout functions, it also possesses a series of functions for sensing, understanding, and intelligently generating information about the supermarket scene, in order to further optimize merchant operations and improve the customer's shopping experience.
[0035] This application proposes to add visual sensors on both sides, a shelf scene perception server, and a series of software algorithms to the existing common smart shopping cart equipment, giving the shopping cart the ability to perceive and understand the scene, and ultimately realizing functions such as shelf data analysis, inventory management, personalized recommendations, and advertising.
[0036] Figure 1 is a flowchart of a shelf scene perception method based on a shopping cart in an embodiment of this application, applied to a shopping cart, including:
[0037] Step 101: As the shopping cart is pushed by the shopper, obtain images of the product displays on each shelf through the visual sensors on the shopping cart;
[0038] Step 102: Identify the shelf scene corresponding to the image. When the shelf scene is an analyzable shelf layout, identify the product information of each product in the image.
[0039] Step 103: Obtain the product display plan, which is obtained after identifying the products on each shelf;
[0040] Step 104: When the number of identified products exceeds a preset threshold, the location information of the shopping cart is diffused to obtain the location interval that matches the predefined shelf location. Combined with the identified product information, identification confidence, and location interval, the area to be updated and the product to be updated in the product display drawing are located. The location interval includes at least 2 locations.
[0041] Step 105: Crop the product to be updated from the area in the image to obtain a small image of the product to be updated;
[0042] Step 106: Compare the small image of the product to be updated with the pre-stored image of the product to be updated. If the similarity is greater than the similarity threshold, the product to be updated is determined to be correct. Record the pending update status of the area to be updated according to the product to be updated. The pending update status record is used to update the pending update area of the product layout drawing.
[0043] Compared with the prior art, this application can identify the shelf scene corresponding to the image captured by the shopping cart, and then identify the product information in the shelf scene. In order to improve the accuracy and efficiency of updating the product display drawing, when the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred and diffused to obtain the location range. Then, combined with the product information, the area to be updated is obtained, and only the area to be updated needs to be updated.
[0044] In step 101, as the shopping cart is pushed by the shopper, images of each shelf display are acquired via visual sensors on the shopping cart.
[0045] In specific embodiments, the visual sensor can be a camera. The shopping cart proposed in this application is a smart shopping cart, which is based on a common smart shopping cart structure (originally equipped with some sensors and computing devices facing the basket area), with the addition of visual sensors (cameras) on both sides. In addition, the shopping cart's hardware can perform positioning based on its own hardware (gyroscope sensor), and can know its location in the supermarket. The shelf scene perception server proposed in this application can communicate with the backend server of the shopping cart computing device, and can send information to the shopping cart device, and can also receive information from the shopping cart.
[0046] Cameras installed on both sides of the shopping cart can take pictures at a certain frequency while shoppers are using the cart, capturing images that can be used for shelf product display analysis. After a series of algorithms, the system can perform reasoning and calculations to analyze the product display and stockout status on the shelves, as well as the shopper's position, purchased items, and dwell time, providing personalized shopping recommendations.
[0047] While the shopping cart device is being used by shoppers, the visual sensors take pictures at configured fixed intervals to capture the current moment.
[0048] Specifically, considering that if the shopping cart remains stationary without being pushed, there is no point in performing unnecessary image processing and calculations, it is necessary to perform difference calculations on the captured images. Specifically, the image subtraction method is used to calculate whether the images captured between two intervals are consistent. If the images are consistent, no further processing is required.
[0049] In one embodiment, the method further includes:
[0050] The system acquires multiple images of each shelf's merchandise display taken at preset intervals using a visual sensor.
[0051] Use image subtraction to determine whether multiple images taken at preset intervals are identical.
[0052] If so, identify the shelf scene corresponding to the image.
[0053] In step 102, the shelf scene corresponding to the image is identified. When the shelf scene is an analyzable shelf layout, the product information of each product in the image is identified.
[0054] Because shopping carts capture various images during use that are unsuitable for shelf display analysis—such as images obstructed by pedestrians, carts too close to the shelves to show good product displays, carts lying horizontally in aisles and not showing shelf displays, and carts too far from the shelves—image analysis is necessary. The specific methods are as follows:
[0055] The captured images are fed into a scene recognition model that can identify at least nine types: "pedestrian occlusion", "object occlusion", "camera being blocked by being too close", "no shelf scene", "shelf display too far away", "shelf display too close", "image too blurry", "shelf display too crooked", and "analyzable shelf display (normal shelf display)". Of course, there may be other scene classifications, which are not limited in this application.
[0056] In one embodiment, identifying the shelf scene corresponding to the image includes:
[0057] The image is input into the scene recognition model, which identifies the corresponding shelf scene. The scene recognition model is used to analyze the shelf scene type.
[0058] The scene recognition model is a classification model built on a deep neural network model.
[0059] Scene recognition models can be trained using a large number of labeled image datasets and deep learning-based training methods.
[0060] In one embodiment, before identifying the product information of each item in the image, the method further includes:
[0061] Perform image correction on the image to obtain the corrected image;
[0062] The product information of each product in the image is identified, including: inputting the corrected image into the product layout detection model, detecting the bounding box of each product in the corrected image. The product layout detection model is a target detection model built on a deep neural network model.
[0063] Based on the bounding box of each product, extract the product thumbnail region from the corrected image;
[0064] The product image region is input into the product recognition model to identify the product information of each product. The product recognition model is a classification model built on a deep neural network model.
[0065] Specifically, the product display detection model is trained using a large dataset of labeled images and based on deep learning methods. A preferred example is the You Only Look Once version 8 (YOLOv8) model (which offers high accuracy and efficiency).
[0066] The product identification model identifies the product information for each item, including its specific Stock Keeping Unit (SKU) or name. It can be trained using a large dataset of labeled images, employing deep learning-based training methods (metric learning-based methods, such as incorporating a triplet loss function). Product identification can utilize either direct classification or computational retrieval methods.
[0067] In one embodiment, image correction includes:
[0068] Binarize the image;
[0069] Perform edge detection on the binarized image (using the Canny or Sobel operator) and extract the horizontal edge information of the binarized image;
[0070] Detect straight lines in the binarized image and count the straight lines corresponding to the shelf panels based on a preset length threshold;
[0071] Based on the statistically obtained angle distribution and main direction of the straight lines, the tilt angle of the binarized image is calculated;
[0072] Based on the calculated tilt angle, the binarized image is rotated so that the binarized image tends to be horizontal according to the layer plate.
[0073] Specifically, various methods can be used to detect straight lines in binarized images; no specific limitations are imposed here. For example, the Hough line transform is a classic method for detecting straight lines or line segments in images. Other methods are given below.
[0074] 1. LSD (Line Segment Detector): LSD is a line segment detection algorithm based on gradient information. It can quickly detect line segments in an image. Its advantages are fast calculation speed, suitable for real-time applications, and the ability to detect relatively short line segments. Its disadvantage is that it is more sensitive to noise.
[0075] 2. EDLines (Edge Drawing Lines): EDLines is a line detection algorithm based on edge drawing. It forms line segments by connecting edge points. Its advantages are high computational efficiency and the ability to detect relatively short line segments. Its disadvantage is that it is more sensitive to the way edge points are connected.
[0076] 3. PPHT (Progressive Probabilistic Hough Transform): PPHT is an improved version of the Hough Transform. It detects lines by progressively accumulating votes, reducing computational cost. Its advantages include lower computational cost, making it suitable for processing larger images; its disadvantage is that its detection accuracy may not be as high as the standard Hough Transform. PPHT is suitable for scenarios requiring large image processing or real-time detection.
[0077] 4. RANSAC (Random Sample Consensus): RANSAC is a model fitting algorithm based on random sampling that can be used to detect straight lines in images. Its advantages include good robustness to noise and outliers; its disadvantages include high computational cost, making it suitable for processing small images or specific regions, and applicable to noisy scenarios.
[0078] 5. HoughP (Probabilistic Hough Transform): HoughP is another improved version of the Hough Transform, which detects lines by randomly sampling edge points. Its advantages are lower computational cost, making it suitable for processing larger images; its disadvantage is that its detection accuracy may not be as high as the standard Hough Transform.
[0079] 6. Deep learning methods, such as using convolutional neural networks (CNNs) for line detection. The advantages are the ability to learn complex image features and high detection accuracy; the disadvantages are the need for large amounts of training data and computational resources.
[0080] 7. LSDNet: LSDNet is a deep learning-based line detection network that combines the traditional LSD algorithm with deep learning techniques. Its advantages include high detection accuracy and the ability to handle complex image scenes. Its disadvantages include the need for large amounts of training data and computational resources.
[0081] 8. FAST Line Detector: FAST is a line detection algorithm based on corner detection. It detects corners and connects them to form lines. Its advantages are high computation speed, making it suitable for real-time applications. Its disadvantage is that it is relatively sensitive to noise.
[0082] 9. GHT (Generalized Hough Transform): GHT is an extended version of the Hough transform, capable of detecting objects of arbitrary shapes, including lines. Its advantage is its ability to detect objects of any shape. Its disadvantage is its high computational cost, making it suitable for processing small images or specific regions.
[0083] In addition, in the embodiments of this application, an image rotation function can be used to rotate the binarized image.
[0084] The following is a preferred embodiment, which can narrow down the product recognition range based on the location information of the shopping cart and the original layout drawing information.
[0085] In one embodiment, the product thumbnail area is input into the product recognition model to identify the product information of each product, including:
[0086] Based on the shopping cart's location information and product layout diagram, the product identification range is determined;
[0087] Input the recognition range and the product thumbnail area into the product recognition model so that the product recognition model can identify each product within the recognition range.
[0088] For example, if the shopping cart is at position (200, 350), and the shelf at that position has products listed as "A, B, C, D, E, F, G" on the shelf layout diagram, then during product identification, the classification or retrieval range can be limited to "A, B, C, D, E, F, G". This range is the product identification range, which can improve the accuracy of product identification.
[0089] In step 103, a product display diagram is obtained, which is obtained after identifying the products on each shelf;
[0090] The shelf scene perception server stores product display drawings. The initial product display drawings are generated by store clerks (or site surveyors) pushing shopping carts to collect images. The shopping cart needs to traverse all shelves, and the image of the shelf captured at each location is basically upright and includes almost all the products on that shelf. Of course, product display drawings can also be obtained from other places, and this application does not impose any restrictions.
[0091] In one embodiment, the method further includes:
[0092] As the shopping cart is pushed by the store clerk, when the shopping cart's positioning information aligns with the predefined shelf locations, the visual sensors on the shopping cart collect images of the product displays corresponding to each shelf location.
[0093] Recognize the images of the product displays on each shelf;
[0094] Based on the identification results, the products on each shelf are labeled;
[0095] Based on the labeled products, create a product layout drawing for each shelf;
[0096] The product layout diagram is sent to the shelf scene perception server.
[0097] During the aforementioned identification process, each item on the shelf can be marked using identification algorithms or manual methods to create a product display diagram, which is then sent to the shelf scene perception server. Subsequent updates involve downloading from the shelf scene perception server, updating, and then uploading again, ensuring the product display diagram on the shelf scene perception server is up-to-date. When each shopping cart is used, it can instantly connect to the shelf scene perception server to retrieve the latest shelf location and product display diagram data. Updates can occur either when the shopping cart is moved and put into use or at scheduled intervals. Simultaneously, the shopping cart device can correlate its own location information with the shelf location.
[0098] In step 104, when the number of identified products exceeds a preset threshold, the location information of the shopping cart is diffused to obtain a point range that matches a predefined shelf location. Combining the identified product information, identification confidence, and point range, the area to be updated and the products to be updated in the product display drawing are located. The point range includes at least two points.
[0099] Specifically, the preset threshold can be determined according to the actual situation. If the preset threshold is too large, the update frequency is low and the resource consumption is low; if the preset threshold is too small, the update frequency is high, but the resource consumption is high. In this embodiment, the preset threshold can be set to 4. Since the location of the shopping cart may not be in the location information recorded during the site survey, it is necessary to perform fuzzy diffusion to find the location recorded during the site survey, that is, to match the predefined shelf location range. If the shopping cart is at coordinates (200, 350), then it can be fuzzily diffused to a location range that contains at least two shelves, and the location range is (150~250, 300~350).
[0100] In one embodiment, by combining the identified product information, identification confidence level, and location range, the area to be updated and the product to be updated in the product display drawing are located, including:
[0101] From the identified product information, determine the products that overlap with the products in the location range and the products that need to be updated;
[0102] If the recognition confidence of each overlapping product is greater than the recognition confidence threshold, the position of the image corresponding to the overlapping product in the product layout drawing is determined as the area to be updated.
[0103] For example, if the three shelf items corresponding to the interval (150-250, 300-350) are arranged as [[A,B,C,D],[E,F,G,H],[I,J,K,L]], and the identified item information is [[E,F,G,M]], then the duplicate items are E, F, and G, and the item to be updated is M. If the identification confidence scores for the three items "E,F,G" all exceed the identification confidence threshold (e.g., 0.8), then the area to be updated in the product display diagram can be located at the position of the image corresponding to the [E,F,G,H] item in the product display diagram.
[0104] In one embodiment, before cropping the item to be updated from the area in the image, the method further includes:
[0105] If the average confidence score of all identified product information is greater than the confidence score threshold, the product to be updated in the image will be cropped out from its current region.
[0106] For example, before the update, the arrangement of goods on the shelf corresponding to the area to be updated is [[A,A,B,B,B,C,C],[D,D,E,E,E,F,F]]. However, the arrangement of goods on the shelf corresponding to the area to be updated is now [[A,A,B,B,B,C,C],[D,D,G,G,G,F,F]]. That is, "goods F" has been replaced with "goods G", where G is the goods to be updated. If: 1) the average confidence score of all goods information is greater than the confidence score threshold (e.g., 0.8), then the "goods G" area on the image is cropped out to obtain several small images of the goods to be updated.
[0107] In step 106, the small image of the product to be updated is compared with the pre-stored image of the product to be updated. When the similarity is greater than the similarity threshold, the product to be updated is determined to be correct. The update status is recorded for the area to be updated according to the product to be updated. The update status record is used to update the update area of the product layout drawing.
[0108] Several small images of the products to be updated are directly compared with pre-stored images of the product G to be updated (e.g., stored in the supermarket's product information management platform or database). The pre-stored images can be understood as the product's identification photos. If the similarity is very high, greater than the similarity threshold (e.g., 0.9), then "product G" is considered to have been correctly identified. At this point, it is confirmed that there is a difference between the area to be updated and the actual product placement. Then, the area to be updated is recorded as [[A,A,B,B,B,C,C,D,D],[E,E,E,G,G,G,G,G]].
[0109] Specifically, the action of updating the area to be updated in the product display drawing based on the status record can be performed directly by the shopping cart. Alternatively, the shopping cart can send the status record to the shelf scene perception server, allowing the shelf scene perception server to perform the update action for the area to be updated. No specific restrictions are imposed here.
[0110] In one embodiment, after recording the pending update status of the area to be updated according to the item to be updated, the method further includes:
[0111] The status records to be updated are sent to the shelf scene perception server so that the shelf scene perception server can update each area to be updated after receiving consistent status records to be updated from more than a preset number of shopping carts.
[0112] To ensure accuracy, the data needs to be accumulated from multiple shopping carts. If at least a preset number (e.g., 3) of shopping carts confirm that the actual product display differs from the actual product record, and the records to be updated are consistent, then the shelf scene perception server will update the display drawing data to [[A,A,B,B,B,C,C,D,D],[E,E,E,G,G,G,G,G]].
[0113] Optionally, the shelf scene perception server can push a confirmation to the store clerk or administrator before actually updating, and then perform the actual update after confirmation.
[0114] In one embodiment, the method further includes:
[0115] The images of the product display on each shelf collected by the vision sensor are used to detect the shelf area, which is the area on the shelf where the product is displayed and sold.
[0116] Based on the product shelf area, obtain the stock shortage level of each product shelf area;
[0117] For product shelf areas where the stock shortage level reaches the preset stock shortage threshold, the stock shortage information is determined by matching and comparing the shopping cart location information, product layout diagram, and identified product information.
[0118] In one embodiment, performing grid region detection on the image to obtain the product grid region includes:
[0119] The corrected image is input into the commodity shelf area detection model, which is a target detection model built on a deep neural network model.
[0120] Specifically, the commodity shelf area detection model can be trained using a large number of labeled image datasets and based on deep learning training methods.
[0121] In one embodiment, determining the stockout level of each merchandise shelf area based on the shelf area includes:
[0122] Based on the bounding box of the product shelf area, crop out the image within the product shelf area;
[0123] The cropped image is input into the product shortage level judgment model, which outputs the shortage level of each product shelf area. The product shortage level judgment model is a classification model built on a deep neural network model.
[0124] Specifically, the product shortage assessment model can be trained using a large dataset of labeled images and a deep learning-based training method.
[0125] In one embodiment, the product shortage determination model is trained using the following steps:
[0126] Obtain an image training set, which includes multiple images of goods on a shelf.
[0127] Each image in the image training set is labeled with a tag representing the product's fullness.
[0128] The tagged images are classified into out-of-stock levels to obtain an out-of-stock level tag set;
[0129] Based on the image training set and the corresponding shortage level label set, a product shortage level judgment model is trained to obtain a well-trained product shortage level judgment model.
[0130] For example, for goods within a shelf, if they fill the entire shelf, the product fullness label is 100%; conversely, if no goods are displayed, the label is 0%. For goods that are neither fully occupied nor completely sold out, appropriate shortage level labels are applied based on manual experience and the estimated number of items sold, such as 12%, 25%, 50%, 75%, etc. After labeling, the shortage level is categorized, creating a shortage level label set. Based on fine-grained requirements, the shortage level can be divided into several levels, ultimately used to determine the category of the product shortage level judgment model. Assuming there are n shortage level levels, the span of each level is 100 / (n-1). Within this span, each shortage level can be classified into one level. Specific, but not limited, examples: If the out-of-stock level is divided into 11 levels, then the span of each out-of-stock level is 10%. Here, the rounding method is used, 54% is divided into 50%, 57% is divided into 60%, and finally a set of out-of-stock level labels of n levels are formed, such as 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 100%.
[0131] Finally, for product sections where the stock shortage reaches a preset threshold, the stock shortage information is determined by matching and comparing the shopping cart location information, product layout diagram, and identified product information. Specifically, the preset stock shortage threshold can be determined based on actual conditions; for example, a threshold of 60% or higher is considered a stock shortage for that product.
[0132] Out-of-stock information includes the location of the out-of-stock item, the item being out of stock, and the degree of out-of-stock status. For example, at this location, the identified item is [A,A,A,B,_,_,B,C,C], and the bounding box of the item grid area exactly covers "B,_,_,B". Therefore, we can determine that the out-of-stock item is B, and the degree of out-of-stock status is the result output by the item out-of-stock status judgment model.
[0133] Out-of-stock information can be sent to the shelf scene perception server so that the server can remind store staff to handle it.
[0134] In addition, in this embodiment of the application, if the bounding box and product information of each product can be identified, shopping recommendations can be made accordingly.
[0135] In one embodiment, the method further includes:
[0136] The shopping cart's location information and the identified product information are sent to the shelf scene perception server, so that the shelf scene perception server can construct a shopping point frequency heat map based on the location information and product information. The shopping point frequency heat map is used to describe the frequency of the shopper's shopping at each shelf point at each time point.
[0137] Download the shopping point frequency heatmap from the shelf scene perception server, and obtain the target shelf points whose frequency is greater than the preset frequency at the current time point from the shopping point frequency heatmap;
[0138] The shopping cart display shows the first type of promotional advertisement, which is an advertisement for product information at the current time and the target shelf location.
[0139] Specifically, the shopping cart's location information and the identified product information are sent to the shelf scene perception server. For example, if products [A,B,C] are captured at point (330, 670), this information can be sent to the shelf scene perception server. The shelf scene perception server can then summarize the paths traveled by all shopping carts used by shoppers throughout the day and the products captured. Based on this, the shelf scene perception server can statistically analyze and construct a heatmap of shopper shopping locations. For example, between 10 am and 11 am, most shoppers enter the shelf area displaying products [A,B,C]. The shopping cart device instantly receives the shopping location frequency heatmap generated by the backend server, and the shopping cart can then push corresponding promotional advertisements to shoppers on the display screen. For example, between 10 am and 11 am, advertisements or promotional coupons for products [A,B,C] can be pushed on the display screen.
[0140] Recommended advertisements may also include promotional coupons, etc.
[0141] In one embodiment, the method further includes:
[0142] The image is input into the promotional sign detection model to obtain the detection bounding box of the promotional signs on the shelf;
[0143] Based on the detection bounding box of the promotional sign, crop out the region small image from the image;
[0144] The small image of the region is fed into the text recognition model to extract the text information on the promotional sign.
[0145] If the extracted text information contains the product's name, the product is identified as a promotional product.
[0146] If the extracted text information does not contain the product name information, the distance matching between the detection bounding box of the promotional sign and the product display drawing is performed based on the position of the detection bounding box of the promotional sign in the image to find the product closest to the promotional sign and identify the product closest to the promotional sign as the promotional product.
[0147] Display information about promotional items on the interactive screen in the shopping cart.
[0148] In one embodiment, the method further includes:
[0149] During the process of the shopping cart being pushed, obtain the real-time location information of the shopping cart and the corresponding shelf location;
[0150] The second type of promotional advertisement is displayed on the shopping cart screen. This type of promotional advertisement targets product information for the obtained shelf locations.
[0151] Specifically, the above solution targets shopping recommendations when the shopping cart is moved at any time. For example, if a shopper is currently standing on a shelf containing products [A, B, C, E, F], and the manufacturer of product B has attractive promotional advertisements, then the promotional advertisements for product B can be displayed on the interactive screen of the shopping cart device, including images, videos, or discount information.
[0152] In one embodiment, the method further includes:
[0153] If the target product already exists in the shopping cart, display promotional ads for related products on the shopping cart's screen.
[0154] Specifically, the associated products of the target product can be predefined. For example, if a shopper has already purchased "black pepper seasoning", then if the identified product belongs to the "fresh food section", then promotional advertisements for various "steaks" can be displayed on the interactive screen of the shopping cart device.
[0155] In one embodiment, the method further includes:
[0156] If the identification results of the product information in multiple images captured over a time series do not change, it is determined that the shopper is pausing.
[0157] Obtain the shelf location corresponding to the current location of the shopping cart;
[0158] The third type of promotional advertisement is displayed on the shopping cart screen. This type of promotional advertisement targets product information for the obtained shelf locations.
[0159] Specifically, the above embodiment addresses the scenario of a shopper pausing in the shopping cart. If over 90% of the images remain almost unchanged across n images, and the product recognition result remains consistent, it can be determined that the shopper is pausing and may be hesitant. In this case, the corresponding products on the shelf at that location can be recommended, and advertising images, videos, or promotional information for those products can be displayed on the interactive screen of the shopping cart device.
[0160] This application also proposes a shelf scene perception device based on a shopping cart, the principle of which is similar to the shelf scene perception method based on a shopping cart, and will not be described in detail here.
[0161] Figure 2 is a schematic diagram of a shelf scene perception device based on a shopping cart in an embodiment of this application, applied to a shopping cart, including:
[0162] The image acquisition module 201 is used to acquire images of each shelf of goods displayed by the visual sensor on the shopping cart as the shopping cart is pushed by the shopper.
[0163] The shelf scene recognition module 202 is used to recognize the shelf scene corresponding to the image. When the shelf scene is an analyzable shelf layout, it identifies the product information of each product in the image.
[0164] The product display layout update module 203 is used to obtain product display layout drawings, which are obtained after identifying the products on each shelf. When the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred to obtain a point range that matches the predefined shelf points. Combining the identified product information, recognition confidence, and point range, the update area and the product to be updated in the product display layout drawing are located. The point range includes at least two points. The product to be updated is cropped from its area in the image to obtain a small image of the product to be updated. The small image of the product to be updated is compared with the pre-stored image of the product to be updated. When the similarity is greater than the similarity threshold, the product to be updated is determined to be correct. The update status is recorded for the update area according to the product to be updated. The update status record is used to update the update area of the product display layout drawing.
[0165] In one embodiment, the product display drawing update module is further used for:
[0166] As the shopping cart is pushed by the store clerk, when the shopping cart's positioning information aligns with the predefined shelf locations, the visual sensors on the shopping cart collect images of the product displays corresponding to each shelf location.
[0167] Recognize the images of the product displays on each shelf;
[0168] Based on the identification results, the products on each shelf are labeled;
[0169] Based on the labeled products, create a product layout drawing for each shelf;
[0170] The product layout diagram is sent to the shelf scene perception server.
[0171] In one embodiment, the shelf scene recognition module is used for:
[0172] The image is input into the scene recognition model, which identifies the corresponding shelf scene. The scene recognition model is used to analyze the shelf scene type.
[0173] The scene recognition model is a classification model built on a deep neural network model.
[0174] In one embodiment, the shelf scene recognition module is further used for:
[0175] Before identifying the product information of each item in the image, the image is corrected to obtain the corrected image;
[0176] The shelf scene recognition module is used to: input the corrected image into the product display detection model, and detect the bounding box of each product in the corrected image. The product display detection model is a target detection model built on a deep neural network model.
[0177] Based on the bounding box of each product, extract the product thumbnail region from the corrected image;
[0178] The product image region is input into the product recognition model to identify the product information of each product. The product recognition model is a classification model built on a deep neural network model.
[0179] In one embodiment, the shelf scene recognition module is further used for:
[0180] Binarize the image;
[0181] Edge detection is performed on the binarized image, and the horizontal edge information of the binarized image is extracted;
[0182] Detect straight lines in the binarized image and count the straight lines corresponding to the shelf panels based on a preset length threshold;
[0183] Based on the statistically obtained angle distribution and main direction of the straight lines, the tilt angle of the binarized image is calculated;
[0184] Based on the calculated tilt angle, the binarized image is rotated so that the binarized image tends to be horizontal according to the layer plate.
[0185] In one embodiment, the shelf scene recognition module is further used for:
[0186] Based on the shopping cart's location information and product layout diagram, the product identification range is determined;
[0187] Input the recognition range and the product thumbnail area into the product recognition model so that the product recognition model can identify each product within the recognition range.
[0188] In one embodiment, the apparatus further includes:
[0189] The product shelf area detection module 204 is used to detect shelf areas in the image and obtain product shelf areas, which are the salable areas of products placed on the shelves.
[0190] The out-of-stock analysis module 205 is used to obtain the out-of-stock level of each product shelf area based on the product shelf area; for product shelf areas where the out-of-stock level reaches the preset out-of-stock threshold, the out-of-stock information is determined by matching and comparing the shopping cart location information, product layout diagram and identified product information.
[0191] In one embodiment, the product display drawing update module 203 is used for:
[0192] From the identified product information, determine the products that overlap with the products in the location range and the products that need to be updated;
[0193] If the recognition confidence of each overlapping product is greater than the recognition confidence threshold, the position of the image corresponding to the overlapping product in the product layout drawing is determined as the area to be updated.
[0194] In one embodiment, the product display drawing update module 203 is used for:
[0195] Before cropping the product to be updated from the image area, if the average recognition confidence of all identified product information is greater than the recognition confidence threshold, the product to be updated from the image area will be cropped.
[0196] In one embodiment, the product display drawing update module 203 is used for:
[0197] After recording the pending status of the area to be updated according to the product to be updated, the pending status record is sent to the shelf scene perception server, so that the shelf scene perception server can update each pending status area after receiving more than a preset number of consistent pending status records sent by shopping carts.
[0198] In one embodiment, the merchandise shelf area detection module is used for:
[0199] The corrected image is input into the commodity shelf area detection model, which is a target detection model built on a deep neural network model.
[0200] In one embodiment, the stockout analysis module is used to:
[0201] Based on the bounding box of the product shelf area, crop out the image within the product shelf area;
[0202] The cropped image is input into the product shortage level judgment model, which outputs the shortage level of each product shelf area. The product shortage level judgment model is a classification model built on a deep neural network model.
[0203] In one embodiment, the product shortage determination model is trained using the following steps:
[0204] Obtain an image training set, which includes multiple images of goods on a shelf.
[0205] Each image in the image training set is labeled with a tag representing the product's fullness.
[0206] The tagged images are classified into out-of-stock levels to obtain an out-of-stock level tag set;
[0207] Based on the image training set and the corresponding shortage level label set, a product shortage level judgment model is trained to obtain a well-trained product shortage level judgment model.
[0208] Referring to Figure 3, which is another structural schematic diagram of the shelf scene sensing device based on a shopping cart in this embodiment of the present application, the shelf scene sensing device based on a shopping cart also includes a shopping recommendation module 301, used for:
[0209] The shopping cart's location information and the identified product information are sent to the shelf scene perception server, so that the shelf scene perception server can construct a shopping point frequency heat map based on the location information and product information. The shopping point frequency heat map is used to describe the frequency of the shopper's shopping at each shelf point at each time point.
[0210] Download the shopping point frequency heatmap from the shelf scene perception server, and obtain the target shelf points whose frequency is greater than the preset frequency at the current time point from the shopping point frequency heatmap;
[0211] The shopping cart display shows the first type of promotional advertisement, which is an advertisement for product information at the current time and the target shelf location.
[0212] In one embodiment, the shopping recommendation module 301 is further configured to:
[0213] During the process of the shopping cart being pushed, obtain the real-time location information of the shopping cart and the corresponding shelf location;
[0214] The second type of promotional advertisement is displayed on the shopping cart screen. This type of promotional advertisement targets product information for the obtained shelf locations.
[0215] In one embodiment, the shopping recommendation module 301 is further configured to:
[0216] If the target product already exists in the shopping cart, display promotional ads for related products on the shopping cart's screen.
[0217] In one embodiment, the shopping recommendation module 301 is further configured to:
[0218] If the identification results of the product information in multiple images captured over a time series do not change, it is determined that the shopper is pausing.
[0219] Obtain the shelf location corresponding to the current location of the shopping cart;
[0220] The third type of promotional advertisement is displayed on the shopping cart screen. This type of promotional advertisement targets product information for the obtained shelf locations.
[0221] This application also proposes a shopping cart-based shelf scene perception system. Figure 4 is a schematic diagram of the structure of the shopping cart-based shelf scene perception system in this application, including:
[0222] The shelf scene perception server 401 and the shopping cart 402 corresponding to the aforementioned device;
[0223] An environment-aware server is used for:
[0224] Receive and store the product layout diagram sent from the shopping cart.
[0225] In one embodiment, the shelf scene perception server is further used for:
[0226] Receive location information and identified product information sent from the shopping cart;
[0227] A shopping location frequency heatmap is constructed based on location information and product information. The shopping location frequency heatmap is used to describe the frequency of shoppers visiting each shelf location at each time point.
[0228] In summary, the methods, apparatus, and systems proposed in this application have the following beneficial effects:
[0229] This application can identify the shelf scene corresponding to the image captured by the shopping cart, and then identify product information in that shelf scene. In order to improve the accuracy and efficiency of updating the product display drawing, when the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred and diffused to obtain the location interval. After that, the product information is combined to obtain the area to be updated, and only the area to be updated needs to be updated. In addition, this application can perform shelf area detection on the image, obtain the degree of stock shortage, and then match and compare the positioning information of the shopping cart, the product display drawing and the identified product information to obtain accurate stock shortage information. The overall shelf perception capability is strong.
[0230] This application embodiment also provides a computer device. Figure 5 is a schematic diagram of the computer device in this application embodiment. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the above-mentioned shelf scene perception method based on shopping cart.
[0231] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described shopping cart-based shelf scene perception method.
[0232] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described shopping cart-based shelf scene perception method.
[0233] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0234] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0235] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0236] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0237] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A shelf scene perception method based on a shopping cart, characterized in that, Applied to shopping carts, including: As the shopping cart is pushed by the shopper, images of each shelf's product display are acquired via visual sensors on the shopping cart. Identify the shelf scene corresponding to the image; when the shelf scene is an analyzable shelf layout, identify the product information of each product in the image. Obtain a product display layout drawing, which is obtained after identifying the products on each shelf; When the number of identified products exceeds a preset threshold, the location information of the shopping cart is diffused to obtain a point range that matches a predefined shelf location. Combining the identified product information and identification confidence level with the point range, the area to be updated and the product to be updated in the product display drawing are located. The point range includes at least two points. The product to be updated is cropped from its location in the image to obtain a small image of the product to be updated; and The small image of the product to be updated is compared with the pre-stored image of the product to be updated. If the similarity is greater than the similarity threshold, the product to be updated is determined to be correct. The update status of the area to be updated is recorded according to the product to be updated. The update status record is used to update the update area of the product layout drawing.
2. The method as described in claim 1, characterized in that, Also includes: The system acquires multiple images of each shelf's merchandise display taken at preset intervals using a visual sensor. Use image subtraction to determine whether multiple images taken at preset intervals are identical. as well as If not, identify the shelf scene corresponding to the most recently captured image.
3. The method as described in claim 1, characterized in that, Also includes: As the shopping cart is pushed by the store clerk, when the shopping cart's positioning information aligns with the predefined shelf locations, the visual sensors on the shopping cart collect images of the product displays corresponding to each shelf location. Recognize the images of the product displays on each shelf; Based on the identification results, the products on each shelf are labeled; Based on the labeled products, create a product layout drawing for each shelf; as well as The product display diagram is sent to the shelf scene perception server.
4. The method as described in claim 1, characterized in that, Identifying the shelf scene corresponding to the image includes: The image is input into a scene recognition model to identify the corresponding shelf scene; the scene recognition model is used to analyze the shelf scene type. The scene recognition model is a classification model built on a deep neural network model.
5. The method as described in claim 1, characterized in that, Before identifying the product information for each item in the image, the process also includes: The image is corrected to obtain the corrected image; Identifying the product information of each product in the image includes: inputting the corrected image into the product layout detection model, and detecting the bounding box of each product in the corrected image. The product layout detection model is a target detection model built on a deep neural network model. Based on the bounding box of each product, extract the product thumbnail region from the corrected image; and The product image region is input into the product recognition model to identify the product information of each product. The product recognition model is a classification model built on a deep neural network model.
6. The method as described in claim 5, characterized in that, Image correction of the image includes: The image is binarized. Edge detection is performed on the binarized image, and the horizontal edge information of the binarized image is extracted; Detect straight lines in the binarized image and count the straight lines corresponding to the shelf panels based on a preset length threshold; Based on the statistically analyzed angle distribution and principal direction of the straight lines, the tilt angle of the binarized image is calculated; and Based on the calculated tilt angle, the binarized image is rotated so that the binarized image tends to be horizontal according to the layer plate.
7. The method as described in claim 5, characterized in that, Input the product thumbnail area into the product recognition model to identify the product information for each product, including: Based on the shopping cart's location information and product layout diagram, the identification range of the products is determined; and The identification range and the product thumbnail area are input into the product identification model so that the product identification model can identify each product within the identification range.
8. The method as described in claim 1, characterized in that, Combining the identified product information and identification confidence level with the location interval, the area to be updated and the product to be updated in the product layout drawing are located, including: From the identified product information, determine the products that overlap with the products in the specified location range and the products to be updated; and If the recognition confidence of each overlapping product is greater than the recognition confidence threshold, the position of the image corresponding to the overlapping product in the product layout drawing is determined as the area to be updated.
9. The method as described in claim 1, characterized in that, Before cropping the product to be updated from its location in the image, the process also includes: If the average confidence level of all identified product information is greater than the confidence level threshold, the product to be updated in the image will be cropped out from its location.
10. The method as described in claim 1, characterized in that, After recording the pending update status of the area to be updated according to the product to be updated, the process also includes: The status records to be updated are sent to the shelf scene perception server so that the shelf scene perception server can update each area to be updated after receiving consistent status records to be updated from more than a preset number of shopping carts.
11. The method as described in claim 1, characterized in that, Also includes: The images of the merchandise display on each shelf collected by the visual sensor are used to detect the shelf area to obtain the merchandise shelf area, which is the area on the shelf where the merchandise is placed and sold. Based on the product shelf areas, the degree of stock shortage in each product shelf area is obtained; as well as For product shelf areas where the stock shortage level reaches the preset stock shortage threshold, the stock shortage information is determined by matching and comparing the shopping cart location information, product layout diagram, and identified product information.
12. The method as described in claim 11, characterized in that, Based on the product shelf areas, the degree of stock shortage for each product shelf area is obtained, including: Based on the bounding box of the product shelf area, the image within the product shelf area is cropped; and The cropped image is input into the product shortage level judgment model, which outputs the shortage level of each product shelf area. The product shortage level judgment model is a classification model built on a deep neural network model.
13. The method as described in claim 12, characterized in that, The following steps were used to train the product shortage assessment model: Obtain an image training set, which includes multiple images of goods photographed on a shelf; Each image in the image training set is labeled with a tag representing the product's fullness. The tagged images are classified into out-of-stock levels to obtain an out-of-stock level tag set; as well as Based on the image training set and the corresponding shortage level label set, a product shortage level judgment model is trained to obtain a well-trained product shortage level judgment model.
14. The method as described in claim 1, characterized in that, Also includes: The location information of the shopping cart and the identified product information are sent to the shelf scene perception server so that the shelf scene perception server can construct a shopping point frequency heat map of the shopper based on the location information and product information. The shopping point frequency heat map is used to describe the frequency of the shopper at each shelf point at each time point. Download the shopping point frequency heatmap from the shelf scene perception server, and obtain the target shelf points whose frequency is greater than the preset frequency at the current time point from the shopping point frequency heatmap; as well as The shopping cart display shows a first type of promotional advertisement, which is an advertisement for product information at the current time and the target shelf location.
15. A shelf scene sensing device based on a shopping cart, characterized in that, Applied to shopping carts, including: The image acquisition module is used to acquire images of each shelf display of goods captured by the visual sensors on the shopping cart as the shopping cart is pushed by the shopper; A shelf scene recognition module is used to identify the shelf scene corresponding to the image, and when the shelf scene is an analyzable shelf layout, to identify the product information of each product in the image; and The product display layout drawing update module is used to obtain product display layout drawings, which are obtained after identifying the products on each shelf. When the number of identified products exceeds a preset threshold, the positioning information of the shopping cart is blurred to obtain a point interval matching a predefined shelf location. Combining the identified product information and identification confidence level with the point interval, the module locates the area to be updated and the product to be updated in the product display layout drawing. The point interval includes at least two points. The product to be updated is cropped from its location in the image to obtain a small image of the product to be updated. The small image of the product to be updated is compared with a pre-stored image of the product to be updated. If the similarity is greater than a similarity threshold, the product to be updated is determined to be correct. The update status of the area to be updated is recorded according to the product to be updated. The update status record is used to update the area to be updated in the product display layout drawing.
16. A shelf scene perception system based on a shopping cart, characterized in that, include: A shelf scene perception server and a shopping cart corresponding to the device described in claim 15; as well as An environment-aware server is used for: Receive and store the product layout diagram sent from the shopping cart.
17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 14.
18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14.
19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 14.