Shopping process monitoring method and apparatus
By combining data from inertial measurement units, weight sensors, and vision sensors, and dynamically adjusting the influence weights, the accuracy of detecting the insertion or removal of goods in smart shopping carts is solved, enabling efficient monitoring of the shopping process and timely judgment of abnormal behaviors.
Patent Information
- Application Number
- PCT/CN2026/095357
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
Existing smart shopping cart systems struggle to accurately detect when items are placed or removed, leading to inaccurate identification of abnormal shopping behavior.
By combining data from inertial measurement units, weight sensors, and vision sensors, and through fusion algorithm logic, the final result and its confidence level of whether a product is placed or removed are dynamically determined. This includes acquiring accelerometer and gyroscope data, weight data, and video frame image data, and dynamically adjusting the influence weights of weight and vision data.
It improves the accuracy of monitoring the shopping process, enabling timely and accurate identification of abnormal shopping behavior and prompting or preventing it.
Smart Images

Figure CN2026095357_12112026_PF_FP_ABST
Abstract
Description
Methods and devices for monitoring the shopping process
[0001] Related applications
[0002] This application claims priority to Chinese patent application No. 202510592884.2, filed on May 8, 2025, and incorporates the entire contents of the aforementioned patent application as part of this application. Technical Field
[0003] This application relates to the field of shopping cart technology, and more particularly to a method and apparatus for monitoring the shopping process. Background Technology
[0004] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.
[0005] When smart shopping carts are deployed in supermarkets and shopping malls, there is a need to prevent abnormal shopping behavior. Since consumers use them independently, some may attempt to shop outside the normal process. Smart shopping carts need to detect, alert, or prevent these behaviors. Accurately determining whether items are being placed or removed from the cart is a crucial aspect of this anti-abnormal behavior function. Many shopping carts on the market rely on single sensors for item placement or removal, such as weight sensors or vision sensors alone. There are also methods that combine weight and vision sensors, but existing methods all have limitations and cannot accurately detect whether items are being placed or removed from the cart. Summary of the Invention
[0006] An embodiment of the first aspect of this application provides a method for monitoring the shopping process, thereby achieving accurate monitoring of the shopping process. The method includes:
[0007] Acquire accelerometer and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor;
[0008] The movement status of the shopping cart is determined based on the accelerometer and gyroscope data.
[0009] Based on the movement status of the shopping cart and the changes in the weight data, determine the weight data identification result and its confidence level of whether the product is put into or taken out of the shopping cart.
[0010] Based on the video frame image data, determine the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart;
[0011] Based on the weight data recognition results and their confidence levels, the video frame image data recognition results and their confidence levels, the dynamically determined weight data influence weights, and the dynamically determined video frame image data influence weights, the final result and its confidence level for determining whether an item has been added to or removed from the shopping cart are fused together.
[0012] An embodiment of the second aspect of this application provides a shopping process monitoring device for accurately monitoring the shopping process. The device includes:
[0013] The acquisition unit is used to acquire accelerometer and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor.
[0014] The state determination unit is used to determine the movement state of the shopping cart based on the accelerometer data and gyroscope data.
[0015] The weight data recognition unit is used to determine the weight data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart based on the movement status of the shopping cart and the changes in the weight data.
[0016] The video frame data recognition unit is used to determine, based on the video frame image data, the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart;
[0017] The final detection unit is used to determine the final result and confidence level of whether an item has been put into or taken out of the shopping cart based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight.
[0018] An embodiment of the third aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for monitoring the shopping process.
[0019] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for monitoring the shopping process.
[0020] An embodiment of the fifth aspect of this application provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the above-described method for monitoring the shopping process.
[0021] In this embodiment, the shopping process monitoring scheme acquires accelerometer and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a vision sensor; determines the shopping cart's movement status based on the accelerometer and gyroscope data; determines the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart based on the changes in the shopping cart's movement status and weight data; determines the video frame image data identification result and its confidence level of whether an item has been placed or removed from the shopping cart based on the video frame image data; and fuses the weight data identification result and its confidence level, the video frame image data identification result and its confidence level, the dynamically determined weight data influence weight, and the video frame image data influence weight to determine the final result and its confidence level of whether an item has been placed or removed from the shopping cart. This enables accurate monitoring of the shopping process, improves the accuracy of monitoring the shopping process, helps to accurately identify abnormal shopping behavior, and promptly alerts or prevents abnormal shopping behavior. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0023] Figure 1 is a flowchart illustrating the monitoring method for the shopping process in an embodiment of this application;
[0024] Figure 2 is a schematic diagram of the system architecture for monitoring the shopping process in an embodiment of this application;
[0025] Figure 3 is a flowchart illustrating the process of establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data in an embodiment of this application.
[0026] Figure 4 is a schematic diagram of the structure of the monitoring device for the shopping process in an embodiment of this application;
[0027] Figure 5 is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.
[0029] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.
[0030] This application proposes a smart hardware-based method for analyzing and judging the shopping process, including a weight sensor installed at the bottom of the cart wheel and a vision sensor installed at the device end. Simultaneously, a unique and novel fusion algorithm is designed to effectively calculate and judge data from the weight sensor and vision sensor, more accurately determining whether goods are placed in or removed from the shopping cart. This proposes a shopping process monitoring scheme, which will be described in detail below.
[0031] Figure 1 is a flowchart illustrating the monitoring method for the shopping process in an embodiment of this application. As shown in Figure 1, the method includes the following steps:
[0032] Step 101: Acquire accelerometer and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor.
[0033] Step 102: Determine the movement status of the shopping cart based on the accelerometer data and gyroscope data;
[0034] Step 103: Based on the movement status of the shopping cart and the changes in the weight data, determine the weight data identification result and its confidence level of whether the product is put into or taken out of the shopping cart.
[0035] Step 104: Based on the video frame image data, determine the video frame image data recognition result and its confidence level of whether the product was put into or taken out of the shopping cart;
[0036] Step 105: Based on the weight data recognition results and their confidence levels, the video frame image data recognition results and their confidence levels, the dynamically determined weight data influence weights, and the dynamically determined video frame image data influence weights, fuse and determine the final result and its confidence level of whether the product has been put into or taken out of the shopping cart.
[0037] As shown in flowchart 1, the shopping process monitoring method provided in this application operates as follows: Accelerometer and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a vision sensor are acquired; the shopping cart's movement state is determined based on the accelerometer and gyroscope data; based on the shopping cart's movement state and changes in the weight data, the weight data identification result and its confidence level for whether an item has been placed or removed from the shopping cart are determined; based on the video frame image data, the video frame image data identification result and its confidence level for whether an item has been placed or removed from the shopping cart are determined; based on the weight data identification result and its confidence level, the video frame image data identification result and its confidence level, dynamically determined weight data influence weight, and dynamically determined video frame image data influence weight, the final result and its confidence level for determining whether an item has been placed or removed from the shopping cart are fused together. This allows for accurate monitoring of the shopping process and improves the accuracy of shopping process monitoring. The following is a detailed description of this shopping process monitoring method.
[0038] The shopping process monitoring method provided in this application monitors the movement state, weight data changes, and visual video frame data of the shopping cart during use, capturing the shopping process. It also proposes a fusion judgment algorithm to effectively determine the information based on data from two sensors, outputting the final information on whether the item has been placed or removed. A detailed description follows.
[0039] To facilitate understanding of the shopping process monitoring method provided in this application embodiment, the system architecture for monitoring the shopping process will first be introduced.
[0040] This application embodiment can add several weight sensors and several vision cameras to the basic hardware of a smart shopping cart to capture the shopper's shopping behavior and accurately determine the process of goods being put in or taken out.
[0041] Figure 2 is a schematic diagram of the system architecture for monitoring the shopping process in an embodiment of this application. As shown in Figure 2, the embodiment of this application may include:
[0042] Barcode scanner 1: A common barcode scanner or barcode reader that can scan and identify barcodes on products, communicate with computing and interactive device 5, and send the identification results to computing and interactive device 5.
[0043] Weight sensor 2: Designed to be installed on the wheels of a shopping cart, it can be flexibly mounted and detached from the support frame of a common shopping cart. The weight sensor has the ability to sense weight, that is, to sense changes in weight and specific values; the weight sensor has communication capabilities, and can communicate and transmit data with computing and interactive devices.
[0044] Visual sensor 3: Installed above and around the shopping cart basket, with one or more sensors; communicates and transmits data with computing and interaction device 5.
[0045] Inertial measurement unit 4 acquires accelerometer and gyroscope data in real time.
[0046] Computing and Interactive Device 5: An electronic device with a touch screen and a computing processing unit with a certain computing power. It is installed on the handle of a shopping cart; it is communicatively connected to a barcode scanner, a weight sensor, and a vision camera (sensor); it can run an operating system and software to display data on the interactive screen; it can process data transmitted from the barcode scanner, weight sensor, and vision camera. The shopping process monitoring method provided in this application embodiment is a method applied to the computing and interactive device.
[0047] The method for monitoring the shopping process provided in this application embodiment may include the following steps.
[0048] 1. Real-time data acquisition. Acquire real-time data from the IMU (Inertial Measurement Unit) sensor, real-time data from the weight sensor, and video frame images;
[0049] 2. Determine whether the shopping cart is in motion based on the non-zero acceleration detected by the accelerometer;
[0050] 3. Detect whether any items have been added or removed based on weight changes;
[0051] 4. Detect whether any items are being placed or removed based on video frame data;
[0052] 5. Integrate the two types of judgment information to comprehensively judge the entry and exit of goods.
[0053] The following section details the methods for monitoring this shopping process.
[0054] 1. Acquire data in real time, i.e., step 101 above.
[0055] The system acquires real-time data from IMU (Inertial Measurement Unit) sensors (accelerometer and gyroscope data), real-time data from weight sensors (weight data), and video frame image data, and transmits the real-time data to the computing and interactive device.
[0056] 2. Based on the data from the accelerometer and gyroscope, determine whether the shopping cart is in motion, i.e., step 102 above.
[0057] Based on the accelerometer and gyroscope data acquired by the IMU sensor, data preprocessing is performed, including noise reduction and calibration. The following steps are then performed on the preprocessed data:
[0058] 1) Calculate the magnitude 'a' of acceleration on the three axes (X, Y, Z) of the accelerometer. total :
[0059] 2) Based on the gyroscope readings, calculate the sum of the absolute values of the angular velocities of each axis to obtain r. total .
[0060] 3) If the magnitude of acceleration a total Equal to or close to the preset modulus threshold (e.g., 9.8 m / s) 2 ), and r total If the sum of absolute values of angular velocities is less than a certain preset threshold (e.g., 0.2 rad / s) and does not change much within a certain time period (within a preset time period) (the change in the magnitude of acceleration is less than or equal to the preset threshold for magnitude change, and the change in the sum of absolute values of angular velocities is less than or equal to the preset threshold for the sum of absolute values of angular velocities), then the shopping cart is considered to be in a stationary state.
[0061] 4) If the magnitude of acceleration a total and r total If all values are greater than the threshold (the magnitude of acceleration is greater than the preset magnitude threshold, and the sum of the absolute values of angular velocities is greater than the preset absolute value threshold) or if there are significant changes within a short period of time (such as 0.5 seconds) (the change in magnitude is greater than the preset magnitude change threshold, and the change in the sum of the absolute values of angular velocities is greater than the preset absolute value change threshold), then the shopping cart is considered to be in motion.
[0062] As can be seen from the above, in one embodiment, determining the movement state of the shopping cart based on the accelerometer data and gyroscope data may include:
[0063] Determine the magnitudes of accelerations on the three axes in the accelerometer data;
[0064] Based on the gyroscope data, determine the sum of the absolute values of the angular velocities of each axis;
[0065] The shopping cart is determined to be stationary when the magnitude of acceleration is less than or equal to a preset magnitude threshold, the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities threshold, the change in magnitude of acceleration within a preset time period is less than or equal to a preset magnitude change threshold, and the change in the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities change threshold.
[0066] As can be seen from the above, in one embodiment, determining the movement state of the shopping cart based on the accelerometer data and gyroscope data may further include:
[0067] The shopping cart is determined to be in motion when the magnitude of acceleration is greater than a preset magnitude threshold, the sum of absolute angular velocities is greater than a preset absolute angular velocities threshold, the change in the magnitude of acceleration is greater than a preset magnitude change threshold within a preset time period, and the change in the sum of absolute angular velocities is greater than a preset absolute angular velocities change threshold.
[0068] In practice, the above-described method of determining the shopping cart's movement status (movement or stationary state) based on the accelerometer and gyroscope data can improve the monitoring accuracy of the subsequent shopping process.
[0069] 3. Detect whether there are goods entering or leaving based on weight changes, i.e., step 103 above.
[0070] If the shopping cart is stationary:
[0071] Judge the changes in weight sensor data (changes in weight data) within a time interval (a preset time interval, such as 1 second):
[0072] If the weight value increases from a stable state (e.g., the weight waveform is a stable waveform), and finally remains stable (within the preset weight threshold range), and the difference is greater than a certain threshold (preset weight threshold, such as 10g), then it is determined that the product has been placed in the container; if the weight value decreases from a stable state, and finally remains stable (within the preset weight threshold range), and the difference is greater than a certain threshold (preset weight threshold, such as 10g), then it is determined that the product has been removed; and at this time, the confidence value Conf_a (confidence level of the weight data recognition result) is set to 1.
[0073] As can be seen from the above, in one embodiment, determining the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart based on the shopping cart's movement status and changes in the weight data may include:
[0074] When the shopping cart is detected to be stationary, the changes in weight data within a preset time interval are detected.
[0075] If, based on the detection results, the weight value changes from a stable state to an increasing state, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being added to the shopping cart and its confidence level are obtained.
[0076] As can be seen from the above, in one embodiment, determining the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart, based on the shopping cart's movement status and changes in the weight data, may further include:
[0077] If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart and its confidence level are obtained.
[0078] In practice, the above-described implementation method, which obtains the weight data of the goods removed from the shopping cart and its confidence level when the shopping cart is detected to be stationary, can further improve the accuracy of shopping process monitoring.
[0079] If the shopping cart is in motion:
[0080] Judge the changes in weight sensor data (changes in weight data) within a time interval (a preset time interval, such as 1 second):
[0081] First, the variance of the values within the time interval is calculated. If the variance of the weight change data is greater than a certain threshold (preset variance threshold), then the weight sensor change data within that time interval is determined to be of the type of drastic fluctuation (it is determined that the weight data change within the preset time interval is of the type of drastic fluctuation), and the use of the weight sensor data for the time interval to determine the entry and exit of goods is abandoned; that is, the confidence value Conf_a is set to 0. If the weight change data is not of the type of drastic fluctuation (if the variance value is less than or equal to the preset variance threshold, it is determined that the weight data change within the preset time interval is not of the type of drastic fluctuation), then the weight change data within a certain time interval is further determined:
[0082] If the weight value increases from a stable state and eventually remains stable with the difference exceeding a certain threshold (preset weight threshold), it is determined that the product has been added to the shopping cart (resulting in weight data recognition of the product being added to the shopping cart); if the weight value decreases from a stable state and eventually remains stable with the difference exceeding a certain threshold (preset weight threshold), it is determined that the product has been removed from the shopping cart (resulting in weight data recognition of the product being removed from the shopping cart).
[0083] The weight data used to determine whether an item is "added" or "removed" (the weight value corresponding to when an item is added to the shopping cart) is fed into the "reliability scoring model" (i.e., the confidence recognition model) for scoring. The score is used as the confidence value Conf_a (confidence of the weight data recognition result) that can be used for judgment, and the value is between [0,1].
[0084] The "Reliability Scoring Model" is a discriminative classification model trained using machine learning methods. It is trained from historically collected datasets of weight sensor changes and their corresponding labels. Ultimately, it can score the weight sensor change data within a given time interval and output a score value between [0,1].
[0085] As can be seen from the above, in one embodiment, determining the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart based on the shopping cart's movement status and changes in the weight data may include:
[0086] When the shopping cart is detected to be in motion, the changes in weight data within a preset time interval are detected.
[0087] Determine the variance value of the weight data within a preset time interval. If the variance value is less than or equal to a preset variance threshold, determine that the change in the weight data within the preset time interval does not belong to the type of drastic fluctuation.
[0088] If the weight data changes within the preset time interval are determined not to be of the type of drastic fluctuation, and if the weight value changes from a stable state to an increasing state and remains stable after a preset time period, and the weight difference between the initial weight value in the stable state and the final weight value in the stable state is greater than the preset weight threshold, then the weight data identification result of the product being added to the shopping cart is obtained.
[0089] Based on the weight value corresponding to the item when it was added to the shopping cart, determine the confidence level of the weight data recognition result for the item added to the shopping cart.
[0090] As can be seen from the above, in one embodiment, determining the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart, based on the shopping cart's movement status and changes in the weight data, further includes:
[0091] If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart is obtained.
[0092] The confidence level of the weight data identification result of the removed shopping cart is determined based on the weight value corresponding to the item being removed from the shopping cart.
[0093] As can be seen from the above, in one embodiment, determining the weight data identification result and its confidence level of whether an item has been placed or removed from the shopping cart, based on the shopping cart's movement status and changes in the weight data, further includes:
[0094] If the variance value is greater than the preset variance threshold, it is determined that the weight data change within the preset time interval is of the type of drastic fluctuation, and the use of the weight data in this time interval for judging the entry and exit of goods is abandoned, and the confidence level is determined to be 0.
[0095] In practice, the above-described implementation method, which obtains the weight data of the goods removed from the shopping cart and its confidence level when the shopping cart is detected to be in motion, can further improve the accuracy of shopping process monitoring.
[0096] As can be seen from the above, in one embodiment, determining the confidence level of the weight data identification result of the removed shopping cart based on the weight value corresponding to the time the item was determined to be removed from the shopping cart may include:
[0097] The confidence recognition model is obtained by inputting the weight value corresponding to the item being removed from the shopping cart into the confidence recognition model, and obtaining the confidence value of the weight data recognition result of the removed shopping cart. The confidence value range is within the range of [0,1]. The confidence recognition model is pre-trained and generated based on the sample data of the relationship between historical weight values and confidence values. The input of the model is the weight value, and the output is the confidence value.
[0098] In practice, the above-described method for determining the confidence level of the weight data identification result of the removed shopping cart can improve the efficiency and accuracy of determining the confidence level.
[0099] 4. Based on the video frame data, detect whether there are any goods entering or leaving, i.e., step 104 above.
[0100] To evaluate video frame data within a time interval (e.g., 1 second):
[0101] The time-series video frame data is fed into the video understanding model, which is built and trained using deep learning methods.
[0102] The video understanding model receives input time-series video segment data, outputs which category the time-series video segment belongs to among "put in", "take out", and "put in and take out simultaneously", and outputs the confidence score of the judgment as the confidence score Conf_b for the put in or take out action.
[0103] The details of the video understanding model are as follows:
[0104] 1) Model Architecture: A transformer-based architecture is used. This patent prioritizes the use of the Swin Transformer 3D model, and on this basis, 1. A dynamic attention mechanism, AdaFocus, is introduced, enabling the model to adaptively focus on key frames and key regions in the video, reducing computational redundancy and improving the ability to recognize complex behaviors; 2. The hierarchical structure of the model is improved, and features from different layers and scales in the latter half of the model are fused to further improve the recognition accuracy of product behaviors at different scales.
[0105] 2) Model pre-training: The improved Swin Transformer 3D model built in step 1) is pre-trained using large-scale video data, including the Kinetics-700 dataset, UCF101, HMDB51, NTU RGB+D dataset, etc.
[0106] 3) Data Acquisition and Labeling: Video data of products being placed in or removed from the shopping cart is collected by a vision sensor installed on the shopping cart. The data can come from laboratory scenarios or scenarios of shopping carts being used in stores. The collected video data is manually labeled, marking each video segment with a duration of about 1 second, and labeling each segment with attributes or tags indicating that it belongs to the product being placed in or removed from the shopping cart, forming a standard dataset for subsequent model fine-tuning training and validation.
[0107] 4) Data Preprocessing and Augmentation: For the labeled video dataset, further frame rate adjustment, resolution scaling, and normalization are performed to adapt to the input requirements of the improved Swin Transformer 3D model. Techniques such as random cropping, flipping, lighting transformation, and random acceleration are used to augment the video data, improving the model's robustness to different lighting conditions, viewpoint changes, and motion speeds.
[0108] 5) Model fine-tuning training: Using labeled video data, train the improved Swin Transformer 3D model built in step 1. Use the basic cross-entropy loss function, AdamW as the optimizer and cosine annealing learning rate scheduler to ensure that the model converges and stabilizes during training.
[0109] As described above, in one embodiment, determining the video frame image data recognition result and its confidence level of whether a product is placed or removed from the shopping cart based on the video frame image data includes: inputting the video frame image data into a video understanding model pre-trained according to the following method to obtain the video frame image data recognition result and its confidence level of whether a product is placed or removed from the shopping cart:
[0110] The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model so that the model can adaptively focus on key frames and key regions in video frame image data; the features of different layers and different scales in the latter half of the model are fused to obtain the improved Swin Transformer 3D model.
[0111] An improved Swin Transformer 3D model was pre-trained using video data larger than a preset size to obtain a pre-trained video understanding model.
[0112] By using historical video frame images of labeled products being added to or removed from the shopping cart, the pre-trained video understanding model is fine-tuned and validated to obtain an optimized video understanding model.
[0113] As can be seen from the above, in one embodiment, the method for monitoring the shopping process may further include:
[0114] The historical video frame image data of labeled products being added to or removed from the shopping cart is preprocessed by adjusting the frame rate, scaling the resolution, and normalizing to obtain preprocessed video frame image data, which is adapted to the input requirements of the improved Swing Transformer 3D model.
[0115] The preprocessed video frame image data is enhanced by random cropping, flipping, lighting transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, viewpoint changes and motion speed, and to obtain enhanced video frame image data.
[0116] 5. Integrate the two sets of judgment information to make a comprehensive judgment on the entry and exit of goods, i.e., step 105 above.
[0117] For decision-making information regarding the entry and exit of goods based on weight sensor data and visual video frame data, two influence weights are configured. The default influence weight for judgments based on weight sensor data is set to α = 0.5, and the default influence weight for judgments based on visual video frame data is set to β = 0.5. In practical applications, the values of α and β can be calculated based on empirical values or using machine learning methods, and ultimately adjusted to other more optimal values, such as α = 0.65 and β = 0.35.
[0118] However, this application further employs a method based on historical data to dynamically calculate better values for α and β, that is, to dynamically calculate them based on the current values of Conf_a and Conf_b. Specifically, it is described as follows:
[0119] Collect a large amount of historical data, such as 100,000 video examples. When an insertion or removal action occurs, record Conf_a and Conf_b at that moment. Also record Conf_a and Conf_b when no action occurs, even if they are not zero. For Conf_a and Conf_b, if their values are greater than 0.8, check whether an action actually occurred. If an action did occur, record it as correct; otherwise, record it as incorrect. Finally, accumulate and calculate the accuracy of the weight sensor judgment and the vision sensor judgment separately. Acc a and Acc b .
[0120] With Acc, which calculates history based on history a and Acc b Subsequently, for each action example, different α and β thresholds can be dynamically set based on the different values of Conf_a and Conf_b. The formula (the model for dynamically determining the influence of weight data on weights) is as follows:
[0121] The model for dynamically determining the influence weights of video frame image data is: β = 1 - α;
[0122] Where α is the weight of the weight data, β is the weight of the video frame image data, and Acc is the weight of the weight of the weight data. a Acc is the accuracy rate for weight data detection based on historical data. b Conf_a represents the detection accuracy of video frame image data determined based on historical data, Conf_b represents the confidence level of the current weight data recognition result, and Conf_a represents the confidence level of the current video frame image data recognition result.
[0123] Specifically, information on the entry and exit of goods is checked on the timeline. For two types of judgment information, once one judgment is completed, the system waits for the other judgment signal, with a maximum wait time of n seconds (e.g., 1 second). If no signal is received after n seconds, the corresponding influence weight is set to 0. For judgment signals that have already been received, the influence weight is set to 1. For example, if the weight sensor receives a signal but the vision sensor has not yet made a judgment signal within 1 second, then Conf_a = 1 and Conf_b = 0.
[0124] In one embodiment, the above-described method for monitoring the shopping process may further include adjusting the confidence level of the video frame image data recognition result or the video frame image data recognition result based on the determination time of the weight data recognition result and the video frame image data recognition result:
[0125] Once either the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds;
[0126] If the preset number of seconds has elapsed and no other identification result has been received, the influence weight value corresponding to the other identification result is adjusted to 0.
[0127] If the waiting time is less than or equal to the preset number of seconds, and another identification result has already been received, the influence weight of the other identification result is 1.
[0128] In practice, this application embodiment can effectively and accurately analyze and judge the process of putting or taking out goods during the shopping process by adjusting different confidence values and influence weights.
[0129] Finally, the final decision confidence Conf is calculated, which is the final result and confidence of determining whether an item has been added to or removed from the shopping cart based on the weight data identification results and their confidence, the video frame image data identification results and their confidence, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight. This includes determining the final result of whether an item has been added to or removed from the shopping cart according to the following fusion model: Conf=λ(α×Conf_a+β×Conf_b);
[0130] Where Conf is the final result, α is the weight of the weight data, β is the weight of the video frame image data, Conf_a is the confidence level of the current weight data recognition result, Conf_b is the confidence level of the current video frame image data recognition result, λ is the degree of influence of confidence, λ=1 / n, and n is the difference between the determination time of the weight data recognition result and the determination time of the video frame image data.
[0131] Furthermore, regarding the confidence level influence λ, for the two judgment information, the shorter the time difference between reception within n seconds, the larger the value of λ, which is set to λ = 1 / n. The point here is that the more synchronized the two judgment signals are, the higher the probability that the action will occur.
[0132] If Conf is greater than the set threshold, it is determined that an item has been put in or taken out.
[0133] In one feasible embodiment, the method for monitoring the shopping process further includes:
[0134] Based on the final results and their confidence levels, abnormal shopping behavior is flagged.
[0135] For example, when the final result is inconsistent with the checkout list, and the confidence level of the final result is greater than or equal to a preset threshold, a strong prompt is issued to indicate abnormal shopping behavior. This strong prompt can be a direct notification to the shopper, prompting them to correct the abnormal shopping behavior. For example, the strong prompt could be a pop-up window on the shopping cart's computing and interaction device 5, a flashing light on the shopping cart, or an audio prompt from the shopping cart.
[0136] When the final result differs from the checkout list, and the confidence level of the final result is below a preset threshold, a weak prompt is used to indicate abnormal shopping behavior. This weak prompt can be sent to store staff via the backend system, reminding them to verify the items against the checkout list during checkout or when the shopping cart leaves the supermarket. Compared to strong prompts, weak prompts do not need to be sent directly to the shopper, thus avoiding impacting the shopper's shopping experience.
[0137] The discrepancy between the final result and the checkout list could be as follows: For example, the final result might show an item added to the cart, but the item is not listed on the checkout list or the quantity is less than the correct quantity; or, the final result might show an item removed from the cart, but the quantity on the checkout list is more than the correct quantity. The correct quantity can be determined based on weight data and / or video frame image data.
[0138] As can be seen from the above, in one embodiment, as shown in Figure 3, the above-mentioned shopping process monitoring method may further include establishing a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data according to the following methods:
[0139] Step 201: Acquire image data from multiple historical video frames;
[0140] Step 202: Based on multiple historical video frame image data, record the confidence level of the historical weight data recognition result when an item is placed or removed, the confidence level of the historical video frame image data recognition result, the confidence level of the historical weight data recognition result that is not 0 when no item is placed or removed, and the confidence level of the historical video frame image data recognition result.
[0141] Step 203: When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, check whether there is actually a product being put in or taken out at this time to obtain the verification result.
[0142] Step 204: Based on the inspection results, determine the accuracy rate of weight data detection and the accuracy rate of video frame image data detection;
[0143] Step 205: Based on the detection accuracy of weight data and the detection accuracy of video frame image data, establish a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data.
[0144] In practice, the specific implementation methods described above for establishing models that dynamically determine the influence weight of weight data and models that dynamically determine the influence weight of video frame image data can further improve the accuracy of shopping process monitoring.
[0145] In summary, the shopping process monitoring method provided in this application effectively combines data from IMU sensors, weight sensors, and vision sensors. By fusing algorithm logic and adjusting different confidence values and influence weights, it can effectively and accurately analyze and judge the process of putting or taking out goods during the shopping process, and can more accurately determine whether goods are put into or taken out of the shopping cart.
[0146] This application also provides a shopping process monitoring device, as described in the following embodiments. Since the principle behind this device is similar to that of the shopping process monitoring method, its implementation can be found in the implementation of the shopping process monitoring method; repeated details will not be elaborated further.
[0147] Figure 4 is a schematic diagram of the structure of the monitoring device for the shopping process in an embodiment of this application. As shown in Figure 4, the device includes:
[0148] Acquisition unit 01 is used to acquire accelerometer data and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor.
[0149] The state determination unit 02 is used to determine the movement state of the shopping cart based on the accelerometer data and gyroscope data.
[0150] The weight data recognition unit 03 is used to determine the weight data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart based on the movement status of the shopping cart and the changes in the weight data.
[0151] The video frame data recognition unit 04 is used to determine the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart based on the video frame image data.
[0152] The final detection unit 05 is used to determine the final result and confidence level of whether the product is put into or taken out of the shopping cart based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight.
[0153] In one embodiment, the state determination unit is specifically used for:
[0154] Determine the magnitudes of accelerations on the three axes in the accelerometer data;
[0155] Based on the gyroscope data, determine the sum of the absolute values of the angular velocities of each axis;
[0156] The shopping cart is determined to be stationary when the magnitude of acceleration is less than or equal to a preset magnitude threshold, the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities threshold, the change in magnitude of acceleration within a preset time period is less than or equal to a preset magnitude change threshold, and the change in the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities change threshold.
[0157] In one embodiment, the state determination unit is further configured to:
[0158] The shopping cart is determined to be in motion when the magnitude of acceleration is greater than a preset magnitude threshold, the sum of absolute angular velocities is greater than a preset absolute angular velocities threshold, the change in the magnitude of acceleration is greater than a preset magnitude change threshold within a preset time period, and the change in the sum of absolute angular velocities is greater than a preset absolute angular velocities change threshold.
[0159] In one embodiment, the weight data identification unit is specifically used for:
[0160] When the shopping cart is detected to be stationary, the changes in weight data within a preset time interval are detected.
[0161] If, based on the detection results, the weight value changes from a stable state to an increasing state, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being added to the shopping cart and its confidence level are obtained.
[0162] In one embodiment, the weight data identification unit is further configured to:
[0163] If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart and its confidence level are obtained.
[0164] In one embodiment, the weight data identification unit is specifically used for:
[0165] When the shopping cart is detected to be in motion, the changes in weight data within a preset time interval are detected.
[0166] Determine the variance value of the weight data within a preset time interval. If the variance value is less than or equal to a preset variance threshold, determine that the change in the weight data within the preset time interval does not belong to the type of drastic fluctuation.
[0167] If the weight data changes within the preset time interval are determined not to be of the type of drastic fluctuation, and if the weight value changes from a stable state to an increasing state and remains stable after a preset time period, and the weight difference between the initial weight value in the stable state and the final weight value in the stable state is greater than the preset weight threshold, then the weight data identification result of the product being added to the shopping cart is obtained.
[0168] Based on the weight value corresponding to the item when it was added to the shopping cart, determine the confidence level of the weight data recognition result for the item added to the shopping cart.
[0169] In one embodiment, the weight data identification unit is further configured to:
[0170] If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart is obtained.
[0171] The confidence level of the weight data identification result of the removed shopping cart is determined based on the weight value corresponding to the item being removed from the shopping cart.
[0172] In one embodiment, the weight data identification unit is further configured to:
[0173] If the variance value is greater than the preset variance threshold, it is determined that the weight data change within the preset time interval is of the type of drastic fluctuation, and the use of the weight data in this time interval for judging the entry and exit of goods is abandoned, and the confidence level is determined to be 0.
[0174] In one embodiment, the video frame data recognition unit is specifically used to: input the video frame image data into a video understanding model pre-trained according to the following method, to obtain the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart:
[0175] The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model so that the model can adaptively focus on key frames and key regions in video frame image data; the features of different layers and different scales in the latter half of the model are fused to obtain the improved Swin Transformer 3D model.
[0176] An improved Swin Transformer 3D model was pre-trained using video data larger than a preset size to obtain a pre-trained video understanding model.
[0177] By using historical video frame images of labeled products being added to or removed from the shopping cart, the pre-trained video understanding model is fine-tuned and validated to obtain an optimized video understanding model.
[0178] In one embodiment, the video frame data recognition unit is further configured to:
[0179] The historical video frame image data of labeled products being added to or removed from the shopping cart is preprocessed by adjusting the frame rate, scaling the resolution, and normalizing to obtain preprocessed video frame image data, which is adapted to the input requirements of the improved Swing Transformer 3D model.
[0180] The preprocessed video frame image data is enhanced by random cropping, flipping, lighting transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, viewpoint changes and motion speed, and to obtain enhanced video frame image data.
[0181] In one embodiment, the above-described shopping process monitoring device may further include: a modeling unit, configured to establish a model for dynamically determining the influence weight of weight data and a model for dynamically determining the influence weight of video frame image data according to the following methods:
[0182] Acquire image data from multiple historical video frames;
[0183] Based on multiple historical video frame image data, the confidence level of the historical weight data recognition result when an item is placed or taken out is recorded, as well as the confidence level of the historical video frame image data recognition result, the confidence level of the historical weight data recognition result that is not 0 when no item is placed or taken out, and the confidence level of the historical video frame image data recognition result.
[0184] When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, the verification result is obtained by checking whether there is actually a product being put in or taken out at this time.
[0185] Based on the inspection results, determine the accuracy rate of weight data detection and the accuracy rate of video frame image data detection;
[0186] Based on the detection accuracy of weight data and the detection accuracy of video frame image data, models are established to dynamically determine the influence weight of weight data and the influence weight of video frame image data.
[0187] In one embodiment, the above-described monitoring device for the shopping process may further include an adjustment unit for adjusting the confidence level of the video frame image data recognition result or the video frame image data recognition result based on the determination time of the weight data recognition result and the video frame image data recognition result:
[0188] Once either the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds;
[0189] If the preset number of seconds has elapsed and no other identification result has been received, the influence weight value corresponding to the other identification result is adjusted to 0.
[0190] If the waiting time is less than or equal to the preset number of seconds, and another identification result has already been received, the influence weight of the other identification result is 1.
[0191] In one embodiment, the model for dynamically determining the influence of weight data on weights is as follows:
[0192] The model for dynamically determining the influence weights of video frame image data is: β = 1 - α;
[0193] Where α is the weight of the weight data, β is the weight of the video frame image data, and Acc is the weight of the weight of the weight data. a Acc is the accuracy rate for weight data detection based on historical data. b Conf_a represents the detection accuracy of video frame image data determined based on historical data, Conf_b represents the confidence level of the current weight data recognition result, and Conf_a represents the confidence level of the current video frame image data recognition result.
[0194] In one embodiment, the final detection unit is specifically used to: determine whether a product has been added to or removed from the shopping cart according to the following fusion model: Conf=λ(α×Conf_a+β×Conf_b);
[0195] Where Conf is the final result, α is the weight of the weight data, β is the weight of the video frame image data, Conf_a is the confidence level of the current weight data recognition result, Conf_b is the confidence level of the current video frame image data recognition result, λ is the degree of influence of confidence, λ=1 / n, and n is the difference between the determination time of the weight data recognition result and the determination time of the video frame image data.
[0196] Based on the aforementioned inventive concept, as shown in FIG5, this application also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the aforementioned method for monitoring the shopping process.
[0197] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for monitoring the shopping process.
[0198] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for monitoring the shopping process.
[0199] In this embodiment, the shopping process monitoring scheme acquires accelerometer and gyroscope data detected by an inertial measurement unit, weight data detected by a weight sensor, and video frame image data detected by a vision sensor; determines the shopping cart's movement status based on the accelerometer and gyroscope data; determines the weight data recognition result and its confidence level of whether an item has been placed or removed from the shopping cart based on the changes in the shopping cart's movement status and weight data; determines the video frame image data recognition result and its confidence level of whether an item has been placed or removed from the shopping cart based on the video frame image data; and fuses the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the video frame image data influence weight to determine the final result and its confidence level of whether an item has been placed or removed from the shopping cart. This enables accurate monitoring of the shopping process and improves the accuracy of shopping process monitoring.
[0200] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0201] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0204] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for monitoring the shopping process, wherein, include: Acquire accelerometer and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor; The movement status of the shopping cart is determined based on the accelerometer and gyroscope data. Based on the movement status of the shopping cart and the changes in the weight data, determine the weight data identification result and its confidence level of whether the product is put into or taken out of the shopping cart. Based on the video frame image data, determine the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart; Based on the weight data recognition results and their confidence levels, the video frame image data recognition results and their confidence levels, the dynamically determined weight data influence weights, and the dynamically determined video frame image data influence weights, the final result and its confidence level for determining whether an item has been added to or removed from the shopping cart are fused together.
2. The method as described in claim 1, wherein, Based on the accelerometer and gyroscope data, the movement status of the shopping cart is determined, including: Determine the magnitudes of accelerations on the three axes in the accelerometer data; Based on the gyroscope data, determine the sum of the absolute values of the angular velocities of each axis; The shopping cart is determined to be stationary when the magnitude of acceleration is less than or equal to a preset magnitude threshold, the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities threshold, the change in magnitude of acceleration within a preset time period is less than or equal to a preset magnitude change threshold, and the change in the sum of absolute angular velocities is less than or equal to a preset absolute angular velocities change threshold.
3. The method as described in claim 2, wherein, Determining the shopping cart's movement status based on the accelerometer and gyroscope data also includes: The shopping cart is determined to be in motion when the magnitude of acceleration is greater than a preset magnitude threshold, the sum of absolute angular velocities is greater than a preset absolute angular velocities threshold, the change in the magnitude of acceleration is greater than a preset magnitude change threshold within a preset time period, and the change in the sum of absolute angular velocities is greater than a preset absolute angular velocities change threshold.
4. The method of claim 1, wherein, Based on the shopping cart's movement and changes in the weight data, the system determines whether an item has been added to or removed from the shopping cart, along with its confidence level, including: When the shopping cart is detected to be stationary, the changes in weight data within a preset time interval are detected. If, based on the detection results, the weight value changes from a stable state to an increasing state, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being added to the shopping cart and its confidence level are obtained.
5. The method of claim 4, wherein, Based on the shopping cart's movement and changes in the weight data, the determination of whether an item was placed into or removed from the shopping cart, along with its confidence level, also includes: If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart and its confidence level are obtained.
6. The method of claim 1, wherein, Based on the shopping cart's movement and changes in the weight data, the system determines whether an item has been added to or removed from the shopping cart, along with its confidence level, including: When the shopping cart is detected to be in motion, the changes in weight data within a preset time interval are detected. Determine the variance value of the weight data within a preset time interval. If the variance value is less than or equal to a preset variance threshold, determine that the change in the weight data within the preset time interval does not belong to the type of drastic fluctuation. If the weight data changes within the preset time interval are determined not to be of the type of drastic fluctuation, and if the weight value changes from a stable state to an increasing state and remains stable after a preset time period, and the weight difference between the initial weight value in the stable state and the final weight value in the stable state is greater than the preset weight threshold, then the weight data identification result of the product being added to the shopping cart is obtained. Based on the weight value corresponding to the item when it was added to the shopping cart, determine the confidence level of the weight data recognition result for the item added to the shopping cart.
7. The method of claim 6, wherein, Based on the shopping cart's movement and changes in the weight data, the determination of whether an item was placed into or removed from the shopping cart, along with its confidence level, also includes: If the weight value changes from a stable state to a decreasing state based on the detection results, remains stable after a preset time, and the weight difference between the initial weight value from the stable state and the final weight value that remains stable is greater than a preset weight threshold, then the weight data identification result of the product being removed from the shopping cart is obtained. The confidence level of the weight data identification result of the removed shopping cart is determined based on the weight value corresponding to the item being removed from the shopping cart.
8. The method of claim 6, wherein, Based on the shopping cart's movement and changes in the weight data, the determination of whether an item was placed into or removed from the shopping cart, along with its confidence level, also includes: If the variance value is greater than the preset variance threshold, it is determined that the weight data change within the preset time interval is of the type of drastic fluctuation, and the use of the weight data in this time interval for judging the entry and exit of goods is abandoned, and the confidence level is determined to be 0.
9. The method of claim 6, wherein, Based on the weight value of the item when it was removed from the shopping cart, determine the confidence level of the weight data identification result of the removed shopping cart, including: The weight value corresponding to the item being removed from the shopping cart is input into the confidence recognition model to obtain the confidence score of the weight data recognition result of the removed shopping cart. The confidence score value is within the range of [0,1]. The confidence recognition model is pre-trained and generated based on the sample data of the relationship between historical weight values and confidence scores.
10. The method of claim 1, wherein, Based on the video frame image data, determining the video frame image data recognition result and its confidence level for whether a product is placed or removed from the shopping cart includes: inputting the video frame image data into a video understanding model pre-trained according to the following method to obtain the video frame image data recognition result and its confidence level for whether a product is placed or removed from the shopping cart: The dynamic attention mechanism AdaFocus is introduced into the Swin Transformer 3D network model so that the model can adaptively focus on key frames and key regions in video frame image data; the features of different layers and different scales in the latter half of the model are fused to obtain the improved Swin Transformer 3D model. An improved Swin Transformer 3D model was pre-trained using video data larger than a preset size to obtain a pre-trained video understanding model. By using historical video frame images of labeled products being added to or removed from the shopping cart, the pre-trained video understanding model is fine-tuned and validated to obtain an optimized video understanding model.
11. The method of claim 10, wherein, Also includes: The historical video frame image data of labeled products being added to or removed from the shopping cart is preprocessed by adjusting the frame rate, scaling the resolution, and normalizing to obtain preprocessed video frame image data, which is adapted to the input requirements of the improved Swing Transformer 3D model. The preprocessed video frame image data is enhanced by random cropping, flipping, lighting transformation and random acceleration techniques to improve the robustness of the model to different lighting conditions, viewpoint changes and motion speed, and to obtain enhanced video frame image data.
12. The method as claimed in any one of claims 1 to 11, wherein, It also includes models for dynamically determining the influence weights of weight data and models for dynamically determining the influence weights of video frame image data, established using the following methods: Acquire image data from multiple historical video frames; Based on multiple historical video frame image data, the confidence level of the historical weight data recognition result when an item is placed or taken out is recorded, as well as the confidence level of the historical video frame image data recognition result, the confidence level of the historical weight data recognition result that is not 0 when no item is placed or taken out, and the confidence level of the historical video frame image data recognition result. When the confidence level of the historical weight data recognition result is greater than the preset weight data recognition result confidence level threshold, and the confidence level of the historical video frame image data recognition result is greater than the preset video frame image data recognition result confidence level threshold, the verification result is obtained by checking whether there is actually a product being put in or taken out at this time. Based on the inspection results, determine the accuracy rate of weight data detection and the accuracy rate of video frame image data detection; Based on the detection accuracy of weight data and the detection accuracy of video frame image data, models are established to dynamically determine the influence weight of weight data and the influence weight of video frame image data.
13. The method as claimed in any one of claims 1 to 11, wherein, This also includes adjusting the confidence level of the video frame image data recognition results or the video frame image data recognition results based on the determination time of the weight data recognition results and the video frame image data recognition results: Once either the weight data recognition result or the video frame image data recognition result is determined, wait for the other recognition result, and wait for a maximum of a preset number of seconds; If the preset number of seconds has elapsed and no other identification result has been received, the influence weight value corresponding to the other identification result is adjusted to 0. If the waiting time is less than or equal to the preset number of seconds, and another identification result has already been received, the influence weight of the other identification result is 1.
14. The method as claimed in any one of claims 1 to 11, wherein, The model that dynamically determines the influence of weight data on weights is: The model for dynamically determining the influence weights of video frame image data is as follows: β=1-α; Where α is the weight of the weight data, β is the weight of the video frame image data, and Acc is the weight of the weight of the weight data. a Acc is the accuracy rate for weight data detection based on historical data. b Conf_a represents the detection accuracy of video frame image data determined based on historical data, Conf_b represents the confidence level of the current weight data recognition result, and Conf_a represents the confidence level of the current video frame image data recognition result.
15. The method as claimed in any one of claims 1 to 11, wherein, Based on the weight data recognition results and their confidence levels, the video frame image data recognition results and their confidence levels, the dynamically determined weight data influence weights, and the dynamically determined video frame image data influence weights, the final result and its confidence level of whether a product has been placed in or removed from the shopping cart are fused together, including: determining the final result of whether a product has been placed in or removed from the shopping cart according to the following fusion model: Conf=λ(α×Conf_a+β×Conf_b); Where Conf is the final result, α is the weight of the weight data, β is the weight of the video frame image data, Conf_a is the confidence level of the current weight data recognition result, Conf_b is the confidence level of the current video frame image data recognition result, λ is the degree of influence of confidence, λ=1 / n, and n is the difference between the determination time of the weight data recognition result and the determination time of the video frame image data.
16. A monitoring device for the shopping process, wherein, include: The acquisition unit is used to acquire accelerometer and gyroscope data detected by the inertial measurement unit, weight data detected by the weight sensor, and video frame image data detected by the vision sensor. The state determination unit is used to determine the movement state of the shopping cart based on the accelerometer data and gyroscope data. The weight data recognition unit is used to determine the weight data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart based on the movement status of the shopping cart and the changes in the weight data. The video frame data recognition unit is used to determine, based on the video frame image data, the video frame image data recognition result and its confidence level of whether the product is put into or taken out of the shopping cart; The final detection unit is used to determine the final result and confidence level of whether an item has been put into or taken out of the shopping cart based on the weight data recognition result and its confidence level, the video frame image data recognition result and its confidence level, the dynamically determined weight data influence weight, and the dynamically determined video frame image data influence weight.
17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the method of any one of claims 1 to 15.
18. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 15.
19. A computer program product, wherein, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 15.
