Method and apparatus for monitoring internal state of shopping cart

Abstract

Disclosed in the present application are a method and apparatus for monitoring an internal state of a shopping cart. The method comprises: for each image data frame, executing operations of detecting whether there is intentional occlusion in a shopping cart: texture information detection step: detecting edge texture information in each image frame, texture number determination step: determining the number of edge textures on the basis of the edge texture information, and intentional occlusion determination step: when the number of edge textures is less than a preset threshold value of the number of edge textures, determining a preliminary result of there being intentional occlusion in the shopping cart; and performing a secondary intentional occlusion confirmation operation on the preliminary result: increasing a collection frequency to a preset collection frequency, and sequentially performing the texture information detection step, the texture number determination step and the intentional occlusion determination step on each collected image frame in a shopping cart basket area, and when the cumulative number of frames in which intentional occlusion occurs within a preset number of seconds reaches a preset threshold value of the number of frames, determining a final result of there being an intentional occlusion state in the shopping cart. The present application can monitor an internal state of a shopping cart in real time.

Description

Methods and devices for monitoring the status inside shopping carts

[0001] cross-application

[0002] This application claims priority to Chinese patent application No. 202510013660.1, filed on January 3, 2025, and incorporates the entire contents of the aforementioned patent application as part of this application. Technical Field

[0003] This application relates to the field of smart shopping cart technology, and in particular to a method and apparatus for monitoring the state inside a shopping cart. Background Technology

[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0005] With the rapid advancements in technologies such as the Internet of Things, artificial intelligence, big data analytics, mobile payments, and smart hardware, the retail industry is undergoing a transformation towards intelligence and digitalization to better meet consumer needs, enhance the shopping experience, and optimize operational efficiency for merchants. In this wave of transformation, innovation in the checkout process is particularly crucial, leading to the emergence of smart shopping carts, which have become a highlight in supermarket environments. These carts aim to provide a self-service shopping experience, simplifying the traditional shopping and payment process by allowing customers to bypass checkout counters and avoid interaction with cashiers, thus significantly improving the convenience and enjoyment of shopping. Simultaneously, for merchants, this reduces the need for human cashiers, thereby lowering operating costs.

[0006] Generally, smart shopping carts have a range of advanced features, including automatic scanning and checkout, navigation and product location, item recognition and weighing, personalized recommendations and advertising, and intelligent prevention of abnormal shopping behavior. Typically, new smart shopping carts are equipped with visual sensors to capture the shopper's shopping process, using this data for subsequent reasoning and calculations to achieve intelligent checkout and intelligent prevention of abnormal shopping behavior.

[0007] However, during shopping cart use, situations that interfere with the visual experience often occur, such as the camera being deliberately obstructed, the seat being open, children sitting in the cart, or stacked goods blocking the camera's view. When these situations occur, the smart checkout and intelligent anti-abnormal shopping behavior functions face greater challenges. Furthermore, the cart's boundary information and whether the cart is nearly full or has only a few items can affect smart checkout and intelligent loss prevention. Real-time calculation and perception of these factors would allow the smart shopping cart's built-in algorithms to achieve better results. Summary of the Invention

[0008] This application provides a method for monitoring the state inside a shopping cart, used to monitor different states inside the shopping cart in real time. The method includes:

[0009] Acquire video frame image data within the shopping cart basket area collected at a first preset time interval;

[0010] For each frame of image data, perform the following operation to detect whether there is intentional occlusion within the shopping cart:

[0011] The steps of texture information detection are as follows: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image;

[0012] The steps for determining the number of textures are as follows: Determine the number of edge textures based on the edge texture information in the image;

[0013] The steps for determining intentional occlusion are as follows: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be an intentional occlusion state in the shopping cart.

[0014] When determining the preliminary results that the shopping cart may be intentionally obscured, perform the following secondary confirmation operation to intentionally obscure the preliminary results:

[0015] The control shortens the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area;

[0016] Each frame of image acquired at the preset acquisition frequency will be sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment.

[0017] When the number of frames that are intentionally occluded within a preset number of seconds reaches a preset frame threshold, the shopping cart is determined to be in a state that may be intentionally occluded.

[0018] This application embodiment also provides a device for monitoring the state inside a shopping cart, used to monitor different states inside the shopping cart in real time. The device includes:

[0019] The acquisition unit is used to acquire video frame image data within the shopping cart basket area collected at a first preset time interval;

[0020] The status monitoring unit performs the following operation for each frame of image data: detecting whether there is intentional occlusion within the shopping cart.

[0021] The steps of texture information detection are as follows: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image;

[0022] The steps for determining the number of textures are as follows: Determine the number of edge textures based on the edge texture information in the image;

[0023] The steps for determining intentional occlusion are as follows: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be an intentional occlusion state in the shopping cart.

[0024] When determining the preliminary results that the shopping cart may be intentionally obscured, perform the following secondary confirmation operation to intentionally obscure the preliminary results:

[0025] The control shortens the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area;

[0026] Each frame of image acquired at the preset acquisition frequency will be sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment.

[0027] When the number of frames that are intentionally occluded within a preset number of seconds reaches a preset frame threshold, the shopping cart is determined to be in a state that may be intentionally occluded.

[0028] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for monitoring the state inside a shopping cart.

[0029] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for monitoring the state inside a shopping cart.

[0030] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for monitoring the state inside a shopping cart.

[0031] In this embodiment, the scheme for monitoring the state inside a shopping cart involves: acquiring video frame image data within the area of ​​the shopping cart basket collected at a first preset time interval; for each frame of image data, performing the following operations to detect whether there is intentional occlusion within the shopping cart: Texture information detection step: For each acquired frame of image data, digital graphics processing is performed to detect edge texture information in the image; Texture quantity determination step: Based on the edge texture information in the image, the number of edge textures is determined; Intentional occlusion judgment step: When the number of edge textures in the image is less than a preset edge texture quantity threshold, a preliminary determination is made that there may be intentional occlusion within the shopping cart. As a result, when determining the preliminary result that the shopping cart may be intentionally obscured, the following secondary confirmation operation for intentional obscuration is performed: the first preset time interval is shortened to the second preset time interval to reach the preset acquisition frequency, and video frame image data within the area of ​​the shopping cart basket is acquired; each frame image acquired at the preset acquisition frequency is sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional obscuration judgment; when the number of frames with intentional obscuration reaches the preset frame threshold within a preset number of seconds, the final result that the shopping cart may be intentionally obscured is determined. This embodiment of the application can monitor the state inside the shopping cart in real time. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0033] Figure 1 is a flowchart illustrating the method for monitoring the status inside a shopping cart in an embodiment of this application;

[0034] Figure 2 is a flowchart illustrating the process of detecting whether there is unintentional obstruction in the shopping cart in an embodiment of this application;

[0035] Figure 3 is a flowchart illustrating the process of detecting the capacity status of goods in a shopping cart in an embodiment of this application;

[0036] Figure 4 is a flowchart illustrating the detection of the state of the shopping cart basket boundary in an embodiment of this application;

[0037] Figure 5 is a schematic diagram of the output results of the basket boundary detection or segmentation model in the embodiment of this application;

[0038] Figure 6 is a schematic diagram of the device for monitoring the state inside a shopping cart in an embodiment of this application;

[0039] Figure 7 is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0041] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0042] This application provides a method for monitoring the state inside a shopping cart. This method can sense different states within the shopping cart and includes the following steps: acquiring video frame image data within the area of ​​the shopping cart basket collected at a first preset time interval; for each frame of image data, performing the following operations to detect different states within the shopping cart: for each frame of image data, performing the following operations to detect intentional and / or unintentional occlusion states within the shopping cart: based on the video frame image data, detecting whether there is intentional occlusion within the shopping cart; based on the video frame image data, detecting whether there is unintentional occlusion within the shopping cart; when no intentional or unintentional occlusion states are detected within the shopping cart, for each frame of image data, performing the following operations to detect the state of the product capacity and / or the state of the basket boundary within the shopping cart: based on the video frame image data, detecting the state of the product capacity within the shopping cart; based on the video frame image data, detecting the state of the basket boundary within the shopping cart; wherein the detected different states within the shopping cart are used to adjust different strategies of the smart shopping cart. The method is described in detail below.

[0043] Figure 1 is a flowchart illustrating the method for monitoring the status inside a shopping cart in an embodiment of this application. As shown in Figure 1, the method includes the following steps:

[0044] Step 10: Acquire video frame image data within the shopping cart basket area collected at a first preset time interval;

[0045] Step 20: For each frame of image data, perform the following operation to detect whether there is intentional occlusion in the shopping cart:

[0046] Step 201: Texture information detection steps: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image;

[0047] Step 202: Determining the number of textures: Based on the edge texture information in the image, determine the number of edge textures;

[0048] Step 203: Determining intentional occlusion: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be intentional occlusion in the shopping cart.

[0049] Step 204: When it is determined that there are preliminary results in the shopping cart that may be intentionally obscured, perform the following secondary confirmation operation on the preliminary results to intentionally obscure them:

[0050] Step 2041: Control the shortening of the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area;

[0051] Step 2042: Perform the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment on each frame of image acquired at the preset acquisition frequency.

[0052] Step 2043: When the number of frames that have been intentionally occluded within a preset number of seconds reaches a preset frame threshold, determine the final result that there may be a state of intentional occlusion in the shopping cart.

[0053] In the method for monitoring the state inside a shopping cart provided in this application embodiment, the shopping cart can be a smart shopping cart. The method operates by: acquiring video frame image data within the area of ​​the shopping cart basket collected at a first preset time interval; for each frame of image data, performing the following operation to detect whether there is intentional occlusion inside the shopping cart: Texture information detection step: For each acquired frame of image data, performing digital graphics processing to detect edge texture information in the image; Texture quantity determination step: Determining the number of edge textures based on the edge texture information in the image; Intentional occlusion judgment step: When the number of edge textures in the image is less than a preset edge texture quantity threshold, determining that there is a possible occlusion inside the shopping cart. Preliminary results of intentional occlusion: When determining the preliminary result that there may be intentional occlusion in the shopping cart, the following secondary confirmation operation for intentional occlusion is performed: the first preset time interval is shortened to the second preset time interval to reach the preset acquisition frequency, and video frame image data within the area of ​​the shopping cart basket is acquired; each frame image acquired at the preset acquisition frequency is sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment; when the cumulative number of frames with intentional occlusion within a preset number of seconds reaches a preset frame threshold, the final result that there may be intentional occlusion in the shopping cart is determined. This embodiment of the application can monitor different states inside the shopping cart in real time. The method for monitoring the state inside the shopping cart is described in detail below.

[0054] This application provides a method for monitoring the state inside a shopping cart. This method, based on a shopping cart equipped with a visual camera, computing, and interactive devices (hereinafter referred to as a smart shopping cart), endows the smart shopping cart with the ability to perceive and judge the state inside the basket in real time. This capability includes capturing images during the use of the smart shopping cart, and then a series of algorithms determining whether there is intentional or unintentional obstruction inside the basket, the capacity of the goods in the basket, and the boundaries of the basket. By perceiving and judging these states, the results are reported to the computer system of the smart shopping cart. The smart checkout and smart anti-abnormal shopping behavior algorithms adopt specific strategies based on the reported actual situation, ultimately achieving better smart shopping results. Some states inside the basket may include:

[0055] 1) Is the view captured by the built-in camera of the smart shopping cart tablet obstructed?

[0056] 2) Whether the child seat structure of the smart shopping cart body is open.

[0057] 3) Does the shopping cart have a child seat section?

[0058] 4) Are there any stacked items in the smart shopping cart basket area that obstruct the camera's view?

[0059] 5) The quantity (capacity) of goods in the smart shopping cart.

[0060] 6) Information such as the edge position of the smart shopping cart basket.

[0061] The perception and judgment of the aforementioned states in this application embodiment can assist in adjusting the intelligent checkout algorithm and intelligent anti-theft algorithm built into the smart shopping cart. That is, this application embodiment can adjust different strategies based on different conditions inside the cart basket, or provide prompts to shoppers. In other words, this application embodiment can endow the smart shopping device with the ability to perceive the state inside the cart basket, thereby assisting other algorithms within the smart shopping cart in performing intelligent checkout and intelligent prevention of abnormal shopping behavior. Specifically, the detected different states inside the smart shopping cart are used to adjust different strategies of the smart shopping cart (intelligent checkout or intelligent prevention of abnormal shopping behavior, etc.). Simultaneously, this application embodiment uses the integration of deep learning, machine learning, and computer vision algorithms to perceive and discover some states inside the smart shopping cart basket during use.

[0062] In terms of hardware, the method for monitoring the status inside a shopping cart provided in this application embodiment may include:

[0063] 1) Visual sensor (image acquisition device): At least one camera with a wide field of view, including but not limited to a large field of view, fisheye camera, panoramic camera, stereo camera, height-mounted camera, etc. Its coverage area includes the area inside the shopping cart basket and a certain distance outside the shopping cart basket.

[0064] 2) Arithmetic unit (an apparatus for executing the method for monitoring the state inside a shopping cart provided in the embodiments of this application, i.e., the apparatus for monitoring the state inside a shopping cart as described below): receives image data transmitted from a visual sensor, i.e. acquires video frame image data within the area of ​​the shopping cart basket collected by the image acquisition device at a first preset time interval, and calls the built-in algorithm of the device (the method for monitoring the state inside a shopping cart provided in the embodiments of this application) to perform inference calculations to obtain the state inside the shopping cart.

[0065] In terms of software, the method for monitoring the status inside a shopping cart provided in this application embodiment may include:

[0066] This application embodiment utilizes a wide-angle camera to periodically acquire video frame data during the shopping process. The data is then processed by computer vision algorithms, including determining intentional occlusion, determining unintentional occlusion, calculating the capacity of goods in the shopping cart, and detecting the edges and status of the shopping cart.

[0067] The general method for monitoring the state inside a shopping cart provided in this application is as follows:

[0068] 1. Acquire image information within the camera's field of view at intervals using a visual sensor;

[0069] 2. Based on the visual image, use digital image processing algorithms to determine whether there is intentional occlusion;

[0070] 3. Based on the visual image, use a recognition model to determine whether there is unintentional occlusion;

[0071] 4. Based on the visual images, use a recognition model to determine the volume of goods inside the vehicle;

[0072] 5. Based on the visual image, use a detection and segmentation model to detect and delineate the boundaries of the basket.

[0073] The specific process of the method for monitoring the status inside a shopping cart provided in this application embodiment is as follows:

[0074] 1. The visual sensor acquires visual image data at time intervals, i.e., step 10 above.

[0075] A visual image is acquired according to the set low-frequency photo capture time interval, i.e., the first preset time interval, such as 5 seconds.

[0076] 2. Based on the visual image, determine whether there is any intentional obstruction, i.e., step 20 above.

[0077] For intentional obstruction: covering the camera lens directly with your hand or other object. The methods and steps for determining obstruction are as follows:

[0078] 1) First, for the acquired single visual image, digital graphics processing is performed to detect the edge texture in the image, i.e., step 201 above. The edge texture detection algorithm uses edge detection algorithms from digital graphics, including but not limited to the Canny edge detection algorithm and the Sobel operator edge detection algorithm. These edge detection algorithms do not need to process the morphological information of the edges; they are only used to calculate and count the number of edge pixels.

[0079] 2) Count the number of edge textures, i.e., step 202 above.

[0080] 3) By setting a corresponding threshold, if the number of edge textures in the image is less than the threshold number, it is determined that the visual sensor may be deliberately occluded, i.e., step 203 above.

[0081] 4) If situation 3) occurs, a second occlusion confirmation is performed, i.e., step 204 above:

[0082] a) Activate the visual sensor to shorten the photo-taking interval to a second preset time interval, such as 16.67 milliseconds or 33.3 milliseconds, to achieve a preset acquisition frequency, such as 60fps or 30fps, for high-frequency shooting, i.e., step 2041 above; that is, in one embodiment, the preset acquisition frequency can be in the range of 30fps-120fps, preferably 60fps or 30fps, and the second preset time interval can be the shortest shooting duration of the camera: 1 second / preset acquisition frequency, that is, the second preset time interval can be in the range of 8.33 milliseconds-33.33 milliseconds, preferably 16.67 milliseconds or 33.3 milliseconds, which can further improve the accuracy of intentional occlusion detection.

[0083] b) Perform the judgments 1), 2), and 3) on each frame of the image, that is, perform the above-mentioned steps of texture information detection, texture quantity determination, and intentional occlusion judgment on each frame of the image acquired at the preset acquisition frequency, i.e., step 2042 above.

[0084] c) When the number of frames that are intentionally obstructed within a preset number of seconds, such as 1 second, reaches a certain threshold (preset frame threshold, such as 30), the final result is that the smart shopping cart may be in a state of intentional obstruction. For example, it is determined that the camera is intentionally obstructed, i.e., step 2043 above.

[0085] 5) If it is determined in 4) that there is intentional occlusion, then report a "intentional occlusion" signal to the system and at the same time restore the frequency of the visual camera to a low frequency (e.g., 5 seconds).

[0086] 3. Based on the visual image, determine whether there is any unintentional occlusion.

[0087] If no "intentional occlusion" is identified in step 2, then the process begins to determine if there is any "unintentional occlusion".

[0088] For unintentional obstructions, such as when a child seat is open, a child sitting on the seat blocking the camera's view, or goods piled up blocking the camera's view, the methods and steps for judgment are as follows:

[0089] 1) First, for each acquired single visual image, it is fed into the "shopping cart status recognition model" to identify whether it is an unintentional occlusion. The "shopping cart status recognition model" is a classification model built on a deep neural network and trained on a manually selected and categorized image dataset. This dataset defines seven categories: "seat panel open," "child sitting on seat panel obstructing view," "stacked goods obstructing view," "empty cart," "slightly full cart (0% to 30% full)," "medium full cart (30% to 60% full)," and "full cart (60% to 100% full). The input to this shopping cart status recognition model can be historical video frame image data, and the output can include unintentional occlusion states.

[0090] 2) If step 1) determines that the unintentional occlusion type is one of the three categories, such as "seat panel is open", "child sitting on seat panel is obstructing", and "piled goods are obstructing", and the confidence level (in the range of [0.6, 1.0]) reaches a certain threshold, then continue to judge the subsequent n video frame images. When at least a certain proportion (e.g., 90%) of the video frame images are judged to be one of the three categories of unintentional occlusion, such as "seat panel is open", "child sitting on seat panel is obstructing", and "piled goods are obstructing", then it is considered that there is an "unintentional occlusion" in the basket. At this time, a signal of "unintentional occlusion" is reported to the system (at this time, the video screen is blocked, and subsequent judgments of the anti-abnormal shopping behavior algorithm logic are not possible).

[0091] As can be seen from the above, in one embodiment, as shown in Figure 2, the method for monitoring the state inside the shopping cart may further include the following step 30: For each frame of image data, perform the following operation to detect whether there is an unintentional occlusion state inside the shopping cart:

[0092] Step 301: Input a single frame image into the shopping cart state recognition model to identify whether it belongs to an unintentional occlusion state; the shopping cart state recognition model is pre-trained and generated based on the relationship samples between historical video frame image data and unintentional occlusion states.

[0093] Step 302: If the result is determined to be one of the unintentional occlusion types and the confidence level reaches the preset confidence threshold, continue to use the shopping cart state recognition model to recognize each subsequent single frame image. When at least a preset proportion of single frame images are recognized as one of the unintentional occlusion types, it is determined that there is an unintentional occlusion state in the shopping cart at this time.

[0094] In practice, the above-described method for detecting unintentional obstructions in the shopping cart can improve the accuracy of detecting unintentional obstructions.

[0095] 4. Determine the volume of goods inside the vehicle based on the visual image.

[0096] If no "intentional obstruction" is detected in step 2, and no "unintentional obstruction" is detected in step 3, the system can be informed of the current capacity of the goods in the shopping cart basket according to the set low-frequency photo interval. (Subsequent algorithms to prevent abnormal shopping behavior can be adjusted based on the capacity of the goods in the shopping cart basket.) The determination method is as follows:

[0097] 1) First, for each acquired single visual image, it is fed into the "shopping cart status recognition model" to identify and determine the current state of the goods in the cart. The input of this shopping cart status recognition model can be historical video frame image data, and the output can include the state of the goods, as detailed in "3. Determining whether there is unintentional occlusion based on the visual image": "empty cart", "small amount of goods in the cart (0% to 30% of the cart's capacity)", "medium amount of goods in the cart (30% to 60% of the cart's capacity)", "full cart (60% to 100% of the cart's capacity)".

[0098] 2) Recognize and judge the video frame images captured n times in succession, that is, repeat step 1) n times.

[0099] 3) For the n recognition results, sort the number of times the four possible results are "empty basket", "small amount of goods in basket", "medium amount of goods in basket" and "full amount of goods in basket". The case with the most times is determined as the current capacity of goods in the basket.

[0100] 4) Report the identification and judgment results to the system (intelligent shopping cart system, which has intelligent shopping strategies, such as intelligent checkout or intelligent prevention of abnormal shopping behavior).

[0101] As can be seen from the above, in one embodiment, as shown in Figure 3, the method for monitoring the state inside a shopping cart may further include the following step 40: when it is detected that there is no intentional obstruction inside the shopping cart, for each frame of image data, the following operation is performed to detect the quantity of goods inside the shopping cart:

[0102] Step 401: Input each single-frame image into the shopping cart capacity recognition model to identify the type of goods in the shopping cart for each single-frame image; the shopping cart capacity recognition model is pre-trained based on the relationship between historical video frame image data and the state of goods capacity.

[0103] Step 402: Sort the frequency of occurrence of the item capacity type in the shopping cart for all single-frame images;

[0104] Step 403: The product capacity type that appears most frequently is used as the current product capacity status in the shopping cart.

[0105] In practice, the above-described method for detecting the quantity of goods in a shopping cart can improve the accuracy of detecting the quantity of goods in a shopping cart.

[0106] 5. Detect and define the boundaries of the bicycle basket based on the visual image.

[0107] If no "intentional obstruction" is identified in step 2, and no "unintentional obstruction" is identified in step 3, then the boundary information of the bicycle basket in the current frame can be reported to the system according to the set low-frequency photo interval. The determination method is as follows:

[0108] 1) First, for the acquired single visual image, it is fed into the "basket boundary detection / segmentation model (basket boundary detection or segmentation model)," which outputs several boundary coordinate points containing the basket boundary, or a basket region mask. The "basket boundary detection / segmentation model" is built using a deep neural network model and trained on a large training set of images of different vehicle models and basket colors. The training task is defined as object detection or image segmentation. If the task is object detection, the training objective is to have the model output at least 6 accurate basket boundary coordinate points; if the task is image segmentation, the training objective is to have the model output an accurate basket region mask. A schematic diagram is shown in Figure 5. Figure 5 is a schematic diagram of the output result of the basket boundary detection or segmentation model in this embodiment. In Figure 5, the 6 points 51 are the basket boundary coordinate points, and the line 52 connecting these points 51 sequentially can enclose the basket region. In Figure 5, the gray plane region 53 (completely overlapping with the basket region) is a schematic diagram of the basket region mask output by the segmentation model.

[0109] 2) In order to obtain more accurate coordinates of the basket boundary or the basket area mask, inference calculations are performed on the video frame images captured n times in a row, i.e., step 1) is repeated n times.

[0110] 3) For the n recognition results, perform fusion. If the output is the coordinates of the bicycle basket boundary, the final result is the average of each coordinate point after removing the maximum and minimum values ​​from the n results. If the output is a mask of the bicycle basket area, the area with n overlaps after overlapping the masks from the n results is taken as the final result.

[0111] 4) After the calculation and judgment are completed, the information of the vehicle basket boundary is reported to the system.

[0112] As can be seen from the above, in one embodiment, as shown in Figure 4, the method for monitoring the state inside a shopping cart may further include the following step 50: when it is detected that there is no intentional occlusion inside the shopping cart, for each frame of image data, the following operation is performed to detect the state of the shopping cart basket boundary:

[0113] Step 501: Input each single frame image into the basket boundary detection or segmentation model, and output several boundary coordinate points containing the basket boundary, or a basket region mask; the basket boundary detection or segmentation model is pre-trained and generated based on the relationship samples between historical video frame image data and several boundary coordinate points containing the basket boundary, or the basket region mask.

[0114] Step 502: Fuse the output results corresponding to all single-frame images to obtain the state of the shopping cart basket boundary.

[0115] In practice, the above-described method for detecting the state of the shopping cart basket boundary can improve the accuracy of detecting the state of the shopping cart basket boundary.

[0116] As can be seen from the above, in one embodiment, the output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary, including:

[0117] If the output consists of several boundary coordinate points, for the coordinate values ​​of the same coordinate point in all single-frame images, the average value after removing the maximum and minimum values ​​will be used as the final coordinate value of that coordinate point, and the final coordinate values ​​of all coordinate points will be used as the final state of the shopping cart basket boundary.

[0118] If the output is a basket area mask, after overlapping the basket area masks of all single-frame images, the boundary of the region with n overlaps is taken as the final state of the smart shopping cart basket boundary, where n is the number of single-frame images contained in the video frame image.

[0119] In practice, the above-described method of fusing the output results corresponding to all single-frame images can improve the accuracy of monitoring the state of the shopping cart basket boundary.

[0120] As can be seen from the above, in one embodiment, the training task of the shopping cart boundary detection or segmentation model is defined as an object detection task or an image segmentation task; the method for monitoring the state inside the shopping cart further includes: if it is defined as an object detection task, the training objective is to make the model output at least six coordinate points of the shopping cart boundary that meet a first preset accuracy threshold; if it is defined as an image segmentation task, the training objective is to make the model output a shopping cart region mask that meets a second preset accuracy threshold.

[0121] In practice, the above-described method for training the basket boundary detection or segmentation model can improve the accuracy of the basket boundary detection or segmentation model.

[0122] In summary, the method for monitoring the status inside a shopping cart provided in this application embodiment achieves the following:

[0123] 1) Pure vision sensors and pure vision algorithms can be used to judge and calculate various complex states in the shopping cart basket.

[0124] 2) Based on computer vision technology, using deep learning, machine learning and artificial intelligence algorithms, it can analyze sensor data in real time and perceive the shopping cart environment.

[0125] 3) Prioritize the various complex states within the basket and design the overall algorithm according to the logic of "no further calculation if there is obstruction" (priority division means: obstruction has the highest priority, and intentional obstruction has the highest priority. When sensing the state inside the vehicle, first determine the obstruction situation (first determine intentional, then unintentional), and then determine the capacity of goods inside the vehicle, the boundary conditions of the basket, etc.).

[0126] 4) For cases of “deliberate occlusion”, the logic of efficient digital image processing and effective high and low frequency image capture can be used to make the judgment, without using complex deep learning algorithms, thereby reducing development costs and computing power consumption.

[0127] 5) For “unintentional occlusion” and “the capacity of goods in the basket”, we innovatively use a classification model method, combined with multiple fusion, sorting or maximum value strategies, to transform a single judgment problem into a more accurate decision problem, which can obtain better judgment results.

[0128] In summary, the beneficial technical effects of the method for monitoring the state inside a shopping cart provided in this application embodiment are:

[0129] 1) This application provides a method for monitoring the state inside a shopping cart, which enables smart shopping devices to sense the state inside the cart. After obtaining the state inside the cart, the smart shopping device's smart checkout algorithm and smart loss prevention algorithm can adjust different strategies according to different conditions inside the cart to obtain better judgment results.

[0130] 2) The shopping cart sensing method provided in this application embodiment allows the same shopping cart tablet to be seamlessly deployed on different ordinary shopping carts, upgrading ordinary shopping carts into smart shopping carts. In other words, the device for monitoring the status inside the shopping cart and the data collection device in this application embodiment can be directly installed on the smart shopping cart, which reduces most of the site survey process.

[0131] This application also provides a device for monitoring the state inside a shopping cart, as described in the following embodiments. Since the principle behind this device is similar to the method for monitoring the state inside a shopping cart, the implementation of this device can refer to the implementation of the method for monitoring the state inside a shopping cart; repeated details will not be elaborated further.

[0132] Figure 6 is a schematic diagram of the device for monitoring the state inside a shopping cart in an embodiment of this application. As shown in Figure 6, the device includes:

[0133] Acquisition unit 01 is used to acquire video frame image data within the shopping cart basket area collected at a first preset time interval;

[0134] The status monitoring unit 02 is used to perform the following operation for each frame of image data: detecting whether there is intentional occlusion in the shopping cart.

[0135] The steps of texture information detection are as follows: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image;

[0136] The steps for determining the number of textures are as follows: Determine the number of edge textures based on the edge texture information in the image;

[0137] The steps for determining intentional occlusion are as follows: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be an intentional occlusion state in the shopping cart.

[0138] When determining the preliminary results that the shopping cart may be intentionally obscured, perform the following secondary confirmation operation to intentionally obscure the preliminary results:

[0139] The control shortens the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area;

[0140] Each frame of image acquired at the preset acquisition frequency will be sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment.

[0141] When the number of frames that are intentionally occluded within a preset number of seconds reaches a preset frame threshold, the shopping cart is determined to be in a state that may be intentionally occluded.

[0142] In one embodiment, the device for monitoring the state inside a shopping cart may further include an unintentional occlusion detection unit, configured to: for each frame of image data, perform the following operation to detect whether there is an unintentional occlusion state inside the shopping cart:

[0143] A single frame image is input into a shopping cart state recognition model to identify whether it belongs to an unintentional occlusion state; the shopping cart state recognition model is pre-trained and generated based on the relationship samples between historical video frame image data and unintentional occlusion states.

[0144] If the result is determined to be one of the unintentional occlusion types, and the confidence level reaches the preset confidence threshold, the shopping cart state recognition model is used to recognize each subsequent single frame image. When at least a preset proportion of single frames are recognized as one of the unintentional occlusion types, it is determined that there is an unintentional occlusion state in the shopping cart at this time.

[0145] In one embodiment, the device for monitoring the state inside a shopping cart may further include a capacity status detection unit, configured to: when it is detected that there is no intentional obstruction inside the shopping cart, perform the following operation to detect the capacity status of the goods inside the shopping cart for each frame of image data:

[0146] Each single-frame image is input into the shopping cart capacity recognition model to identify the type of goods in the shopping cart for each single-frame image; the shopping cart capacity recognition model is pre-trained based on the relationship between historical video frame image data and the state of goods capacity.

[0147] Sort the frequency of occurrence of the item capacity type in the shopping cart in all single-frame images;

[0148] The most frequently occurring product capacity type will be used as the current product capacity status in the shopping cart.

[0149] In one embodiment, the device for monitoring the state inside a shopping cart may further include a boundary state detection unit, configured to: when it is detected that there is no intentional occlusion inside the shopping cart, perform the following operation to detect the state of the shopping cart basket boundary for each frame of image data:

[0150] Each single frame image is input into the basket boundary detection or segmentation model, and the output includes several boundary coordinate points containing the basket boundary, or a basket region mask; the basket boundary detection or segmentation model is pre-trained and generated based on the relationship samples between historical video frame image data and several boundary coordinate points containing the basket boundary, or a basket region mask.

[0151] The output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary.

[0152] In one embodiment, the output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary, including:

[0153] If the output consists of several boundary coordinate points, for the coordinate values ​​of the same coordinate point in all single-frame images, the average value after removing the maximum and minimum values ​​will be used as the final coordinate value of that coordinate point, and the final coordinate values ​​of all coordinate points will be used as the final state of the shopping cart basket boundary.

[0154] If the output is a basket area mask, after overlapping the basket area masks of all single-frame images, the boundary of the region with n overlaps is taken as the final state of the shopping cart basket boundary, where n is the number of single-frame images contained in the video frame image.

[0155] In one embodiment, the training task of the shopping cart boundary detection or segmentation model is defined as an object detection task or an image segmentation task; the method for monitoring the state inside the shopping cart further includes: if it is defined as an object detection task, the training objective is to make the model output at least six coordinate points of the shopping cart boundary that meet a first preset accuracy threshold; if it is defined as an image segmentation task, the training objective is to make the model output a shopping cart region mask that meets a second preset accuracy threshold.

[0156] Based on the aforementioned inventive concept, Figure 7 is a schematic diagram of a computer device structure according to an embodiment of this application. As shown in Figure 7, this application also proposes a computer device 600, including a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 630, it implements the aforementioned method for monitoring the status inside a shopping cart.

[0157] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for monitoring the state inside a shopping cart.

[0158] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for monitoring the state inside a shopping cart.

[0159] In this embodiment, the scheme for monitoring the state inside a shopping cart involves: acquiring video frame image data within the area of ​​the shopping cart basket collected at a first preset time interval; for each frame of image data, performing the following operations to detect whether there is intentional occlusion within the shopping cart: Texture information detection step: For each acquired frame of image data, digital graphics processing is performed to detect edge texture information in the image; Texture quantity determination step: Based on the edge texture information in the image, the number of edge textures is determined; Intentional occlusion judgment step: When the number of edge textures in the image is less than a preset edge texture quantity threshold, a preliminary determination is made that there may be intentional occlusion within the shopping cart. As a result, when determining the preliminary result that the shopping cart may be intentionally obscured, the following secondary confirmation operation for intentional obscuration is performed: the first preset time interval is shortened to the second preset time interval to reach the preset acquisition frequency, and video frame image data within the area of ​​the shopping cart basket is acquired; each frame image acquired at the preset acquisition frequency is sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional obscuration judgment; when the number of frames with intentional obscuration reaches the preset frame threshold within a preset number of seconds, the final result that the shopping cart may be intentionally obscured is determined. This embodiment of the application can monitor the state inside the shopping cart in real time.

[0160] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0164] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for monitoring the state inside a shopping cart, characterized in that, include: Acquire video frame image data within the shopping cart basket area collected at a first preset time interval; For each frame of image data, perform the following operation to detect whether there is intentional occlusion within the shopping cart: The steps of texture information detection are as follows: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image; The steps for determining the number of textures are as follows: Determine the number of edge textures based on the edge texture information in the image; The steps for determining intentional occlusion are as follows: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be an intentional occlusion state in the shopping cart. When determining the preliminary results that the shopping cart may be intentionally obscured, perform the following secondary confirmation operation to intentionally obscure the preliminary results: The control shortens the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area; Each frame of image acquired at the preset acquisition frequency will be sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment. When the number of frames that are intentionally occluded within a preset number of seconds reaches a preset frame threshold, the shopping cart is determined to be in a state that may be intentionally occluded.

2. The method as described in claim 1, characterized in that, The preset acquisition frequency ranges from 30fps to 120fps, and the second preset time interval is the shortest shooting duration of the camera: 1 second / preset acquisition frequency.

3. The method as described in claim 1, characterized in that, Also includes: For each frame of image data, perform the following operation to detect whether there is unintentional occlusion in the shopping cart: A single frame image is input into a shopping cart state recognition model to identify whether it belongs to an unintentional occlusion state; the shopping cart state recognition model is pre-trained and generated based on the relationship samples between historical video frame image data and unintentional occlusion states. If the result is determined to be one of the unintentional occlusion types, and the confidence level reaches the preset confidence threshold, the shopping cart state recognition model is used to recognize each subsequent single frame image. When at least a preset proportion of single frames are recognized as one of the unintentional occlusion types, it is determined that there is an unintentional occlusion state in the shopping cart at this time.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: When no intentional obstruction is detected in the shopping cart, the following operation is performed to detect the quantity of items in the shopping cart for each frame of image data: Each single-frame image is input into the shopping cart capacity recognition model to identify the type of goods in the shopping cart for each single-frame image; the shopping cart capacity recognition model is pre-trained based on the relationship between historical video frame image data and the state of goods capacity. Sort the frequency of occurrence of the item capacity type in the shopping cart in all single-frame images; The most frequently occurring product capacity type will be used as the current product capacity status in the shopping cart.

5. The method according to any one of claims 1 to 3, characterized in that, Also includes: When no intentional occlusion is detected in the shopping cart, for each frame of image data, the following operation is performed to detect the state of the shopping cart basket boundary: Each single frame image is input into the basket boundary detection or segmentation model, and the output includes several boundary coordinate points containing the basket boundary, or a basket region mask; the basket boundary detection or segmentation model is pre-trained and generated based on the relationship samples between historical video frame image data and several boundary coordinate points containing the basket boundary, or a basket region mask. The output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary.

6. The method as described in claim 5, characterized in that, The output results corresponding to all single-frame images are fused to obtain the state of the shopping cart basket boundary, including: If the output consists of several boundary coordinate points, for the coordinate values ​​of the same coordinate point in all single-frame images, the average value after removing the maximum and minimum values ​​will be used as the final coordinate value of that coordinate point, and the final coordinate values ​​of all coordinate points will be used as the final state of the shopping cart basket boundary. If the output is a basket area mask, after overlapping the basket area masks of all single-frame images, the boundary of the region with n overlaps is taken as the final state of the shopping cart basket boundary, where n is the number of single-frame images contained in the video frame image.

7. The method as described in claim 5, characterized in that, The training task of the shopping cart boundary detection or segmentation model is defined as an object detection task or an image segmentation task; the method for monitoring the state inside the shopping cart further includes: if it is defined as an object detection task, the training objective is to make the model output at least six coordinate points of the shopping cart boundary that meet a first preset accuracy threshold; if it is defined as an image segmentation task, the training objective is to make the model output a shopping cart region mask that meets a second preset accuracy threshold.

8. A device for monitoring the state inside a shopping cart, characterized in that, include: The acquisition unit is used to acquire video frame image data within the shopping cart basket area collected at a first preset time interval; The status monitoring unit performs the following operation for each frame of image data: detecting whether there is intentional occlusion within the shopping cart. The steps of texture information detection are as follows: For each frame of image data acquired, digital graphics processing is performed to detect the edge texture information in the image; The steps for determining the number of textures are as follows: Determine the number of edge textures based on the edge texture information in the image; The steps for determining intentional occlusion are as follows: When the number of edge textures in the image is less than a preset threshold, a preliminary result is determined that there may be an intentional occlusion state in the shopping cart. When determining the preliminary results that the shopping cart may be intentionally obscured, perform the following secondary confirmation operation to intentionally obscure the preliminary results: The control shortens the first preset time interval to the second preset time interval to achieve the preset acquisition frequency and acquire video frame image data within the shopping cart basket area; Each frame of image acquired at the preset acquisition frequency will be sequentially subjected to the above steps of texture information detection, texture quantity determination, and intentional occlusion judgment. When the number of frames that are intentionally occluded within a preset number of seconds reaches a preset frame threshold, the shopping cart is determined to be in a state that may be intentionally occluded.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images