Settlement method for item in vending cabinet based on ai recognition
Patent Information
- Application Number
- PCT/CN2026/071847
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-18
- Filing Date
- 2026-01-12
- Publication Date
- 2026-10-22
Smart Images
Figure CN2026071847_22102026_PF_FP_ABST
Abstract
Description
AI-based payment method for vending machine items Technical Field
[0001] This invention relates to the field of image recognition technology, specifically a settlement method for AI recognition of items in vending machines. Background Technology
[0002] In the field of smart vending machines, accurately identifying product information and constructing its 3D model is crucial for improving user experience and optimizing inventory management. AI-powered item recognition in vending machines primarily relies on cameras capturing continuous images of user purchases to form a video stream, from which items are then identified for sale and processed. However, in existing technologies, items are easily obscured when a user takes them, resulting in only partial images being recognized. This affects the accuracy of item identification and consequently, the efficiency of processing payments.
[0003] Therefore, we propose a settlement method based on AI recognition of items in vending machines to solve the above problems. Summary of the Invention
[0004] The purpose of this invention is to provide a payment method for AI-based item recognition in vending machines, in order to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a settlement method for AI recognition of items in a vending machine, the method comprising the following steps:
[0006] The system acquires a video stream of a user purchasing items from a vending machine, and extracts multiple images of the items for sale from the video stream to obtain an item image set, wherein the item image set includes at least two images of the items from different angles.
[0007] Local feature information of items for sale is extracted from the item image set; local item models corresponding to the items for sale are constructed based on the local feature information.
[0008] Collect reference image information of multiple reference items in the vending machine, and construct reference image models corresponding to the reference items based on the reference image information;
[0009] The local item model is compared with multiple reference image models, and the reference image model that meets the preset conditions is extracted. The reference item corresponding to the reference image model is used as the target item corresponding to the item to be sold.
[0010] Obtain the item information and corresponding price of the target item in the video stream; settle the bill for the item to be sold based on the price and item information of the target item.
[0011] Preferably, the steps of acquiring a video stream of a user purchasing items from a vending machine, and extracting multiple images of the items for sale from the video stream to obtain an item image set, wherein the item image set includes at least two images of the items from different angles, include:
[0012] Image acquisition points are set for the corresponding vending machine. Based on the image acquisition points, the entire process of the user purchasing items from the vending machine is captured as a video stream, and a continuous image sequence containing the movement information of the items to be sold is obtained.
[0013] The motion trajectory of the item to be sold is analyzed from the video stream to determine the initial position of the motion trajectory; an image of the item to be sold is acquired at the initial position and recorded as the first item image;
[0014] Feature points are labeled on the items for sale in the first item image to generate an initial feature point set;
[0015] The contour lines of the items to be sold at multiple locations are sequentially collected along the motion trajectory to form a dynamic contour line sequence;
[0016] The feature point matching comparison between the currently acquired contour line and the contour line at the previous position is performed. Video frames that meet the preset conditions are extracted from the video stream, and the corresponding images of the items to be sold are captured and recorded as the second item image.
[0017] Repeat the contour acquisition and comparison steps until the image of the entire motion trajectory is captured, and obtain the second object image from multiple perspectives;
[0018] The first item image is associated with and stored with multiple second item images to form an item image set containing at least two item images from different angles.
[0019] Preferably, the step of performing feature point matching and comparison between the currently acquired contour line and the contour line at the previous position, extracting video frames that meet preset conditions from the video stream, and cropping the corresponding image of the item to be sold, denoted as the second item image, includes:
[0020] Extract the outline of the item to be sold from the first item image to obtain the initial outline, which is denoted as the first outline;
[0021] Multiple refresh points are set based on the first contour line;
[0022] Images from multiple locations are sequentially acquired along the movement trajectory of the items to be sold, resulting in a sequence of images of the items to be selected.
[0023] Extract the outline of the item to be sold corresponding to each candidate item image to form a sequence of outlines to be compared; based on the sequence of outlines to be compared, perform image cropping to obtain the second item image.
[0024] Preferably, the step of cropping the image based on the sequence of contour lines to be compared to obtain the second item image includes:
[0025] Obtain the current contour line to be compared, and compare the feature points of the current contour line to be compared with the previously captured contour line.
[0026] When a new feature point is detected and no new feature point disappears, continue to acquire images at the next location;
[0027] When a new feature point is detected and at the same time the new feature point disappears, the image cropping condition is triggered.
[0028] The number of new feature points appearing in each candidate item image when the cropping condition is met is counted. The candidate item image with the highest frequency of new feature points is selected for cropping to obtain the second item image.
[0029] Preferably, the step of constructing a local item model corresponding to the item for sale based on local feature information includes:
[0030] Extract local feature points from each item image in the item image set and the relative positional relationships between these local feature points;
[0031] Obtain descriptions of feature points in two consecutively cropped images, perform feature point matching between adjacent images, and identify and extract overlapping feature points;
[0032] The relative pose between adjacent images is predicted based on overlapping feature points. Three-dimensional feature points are recovered based on overlapping feature points and relative pose, and the three-dimensional feature points are aligned to a unified coordinate system to generate a local three-dimensional model of the object.
[0033] Preferably, the step of collecting reference image information of multiple reference items in the vending machine and constructing a reference image model corresponding to the reference items based on the reference image information includes:
[0034] Obtain reference image information of each reference item in the vending machine at different placement angles;
[0035] The reference image information of each reference object at different placement angles is sequentially input into the initial image model to gradually train the initial image model iteratively, so as to obtain the reference image model corresponding to each reference object.
[0036] Preferably, the step of comparing the local item model with multiple reference image models, extracting the reference image model that meets preset conditions, and using the reference item corresponding to the reference image model as the target item corresponding to the item to be sold includes:
[0037] Identify and label the feature points of the object in the local object model, and identify and label the reference feature points in the reference image model;
[0038] The object feature points are superimposed with reference feature points, and the reference image model containing the reference feature points that meet the preset conditions of superposition is used as the reference image model corresponding to the object feature points.
[0039] Preferably, the step of aligning the object feature points with reference feature points and using the reference image model containing the reference feature points that meet preset conditions as the reference image model corresponding to the object feature points includes:
[0040] Obtain item feature points and reference feature points, wherein the item feature points include a first item feature point and a second item feature point, and the reference feature points include a first reference feature point corresponding to the first item feature point and a second reference feature point corresponding to the second item feature point;
[0041] The first item feature point is overlapped with the first reference feature point in multiple reference image models, and the first overlap degree corresponding to each reference image model is calculated.
[0042] Extract the reference image model corresponding to the first degree of overlap that meets the preset first degree of overlap threshold, and obtain a pre-selected reference image model set;
[0043] Determine whether the number of preselected reference image models in the preselected reference image model set is 1;
[0044] If the value is 1, then the pre-selected reference image model is used as the reference image model corresponding to the feature points of the item;
[0045] If it is not 1, then the second item feature point is compared with the second reference feature point of multiple pre-selected reference image models in the pre-selected reference image model set, the second overlap degree corresponding to each pre-selected reference image model is calculated, and the pre-selected reference image model corresponding to the second overlap degree that meets the preset second overlap degree threshold is used as the reference image model corresponding to the item feature point.
[0046] Preferably, the step of acquiring the item information of the target item in the video stream and the price corresponding to the target item; and settling the payment for the item to be sold based on the price and item information of the target item includes:
[0047] Obtain information on the types of target items appearing in the video stream and identify the quantity of each type of target item;
[0048] The specific quantity of each type of target item is counted, and the quantity of the target items and the total number of target item types are used as the item information of the target items;
[0049] For each identified target item, obtain its corresponding price information; based on the obtained price information, calculate the total price of similar target items;
[0050] The system identifies all types of target items and aggregates the total price of each type to complete the settlement of the items for sale.
[0051] Compared with the prior art, the beneficial effects of the present invention are:
[0052] Through a dynamic contour line comparison mechanism, video frames that meet preset conditions are dynamically captured to form multi-view object images. The precise capture of changes in the features of moving objects, through the dual judgment conditions of the appearance and disappearance of feature points, ensures the accuracy of the timing of image capture and effectively improves the integrity and feature recognition of object image acquisition.
[0053] By extracting multiple images of items from different angles from the video stream, a local item model is constructed. The local item model is then compared with multiple reference image models, and reference image models whose similarity meets preset conditions are extracted. The reference items corresponding to the reference image models are used as the target items corresponding to the items to be sold. Through multi-model comparison, items in the vending machine can be accurately identified. Through automated identification and settlement processes, the settlement efficiency of the vending machine can be significantly improved. Attached Figure Description
[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] Example: Please refer to Figure 1. The present invention provides a settlement method technical solution for AI recognition of items in vending machines, including the following steps:
[0058] S1: Obtain the video stream of the user purchasing items from the vending machine, and extract multiple images of the items for sale from the video stream to obtain an item image set, wherein the item image set includes at least two images of the items from different angles.
[0059] The steps of acquiring a video stream of a user purchasing items from a vending machine, and extracting multiple images of the items for sale from the video stream to obtain an item image set, wherein the item image set includes at least two images of the items from different angles, include:
[0060] Image acquisition points are set for each vending machine. A video stream of the entire process of a user purchasing items from the vending machine is captured based on these acquisition points, obtaining a continuous image sequence containing motion information of the items for sale. The motion trajectory of the items for sale is analyzed from the video stream to determine the initial position of the trajectory. An image of the item for sale is acquired at the initial position and designated as the first item image. Feature points are calibrated on the items in the first item image to generate an initial feature point set. Contour lines of the items for sale are sequentially acquired at multiple positions along the motion trajectory to form a dynamic contour line sequence. The currently acquired contour line is compared with the contour line at the previous position using feature point matching. Video frames that meet preset conditions are extracted from the video stream, and the corresponding item images are captured and designated as the second item images. The contour line acquisition and comparison steps are repeated until the image capture of the entire motion trajectory is completed, obtaining second item images from multiple perspectives. The first item images and multiple second item images are associated and stored to form an item image set containing at least two item images from different angles.
[0061] Specifically, for the pre-set image acquisition points of the vending machine, these acquisition points are distributed at predetermined locations around the vending machine to cover different viewing angles of the items for sale. The entire video stream of a user purchasing items from the vending machine is captured through these acquisition points, and a continuous image sequence containing motion information of the items for sale is extracted. This motion information includes at least tracking the movement trajectory of the items, providing a high-quality data foundation for subsequent applications such as item recognition and behavior analysis. The movement trajectory of the items for sale is analyzed from the video stream, and the initial segment position of the trajectory is determined. A first image of the item is acquired at the initial position for subsequent determination of the contour lines at other positions. The contour lines of the items for sale at multiple positions are sequentially acquired along the movement trajectory, forming a dynamic contour line sequence. The currently acquired contour line is compared with the contour line at the previous position using feature point matching, and the following determination is performed: when a new contour line is detected... When feature points are added and some feature points disappear, it is determined that a preset condition is met; the video frame position corresponding to the preset condition is recorded; based on the determination result, the video frame that meets the preset condition is extracted from the video stream, and the corresponding image of the item to be sold is captured to obtain the second item image; the first item image and multiple second item images are associated and stored to form an image set containing at least two item images from different angles; the image set satisfies the following conditions: it contains at least one frontal view image of the initial position; it contains at least one side / top view image in the motion trajectory; there is a verifiable feature point correspondence between each image; through motion trajectory tracking and dynamic contour line analysis technology, the automated acquisition of multi-view images of items in the vending machine is realized, and the accuracy of image capture timing is ensured by the feature point change determination mechanism, which significantly improves the integrity of item image acquisition and recognition reliability.
[0062] The steps of performing feature point matching and comparison between the currently acquired contour line and the contour line at the previous position, extracting video frames that meet preset conditions from the video stream, and cropping the corresponding image of the item to be sold, denoted as the second item image, include: extracting the contour line of the item to be sold from the first item image to obtain an initial contour line, denoted as the first contour line; setting multiple refresh points based on the first contour line; sequentially acquiring images at multiple positions along the movement trajectory of the item to be sold to obtain a sequence of candidate item images; extracting the contour line of the item to be sold corresponding to each candidate item image to form a sequence of contour lines to be compared; and cropping the image based on the sequence of contour lines to be compared to obtain the second item image.
[0063] The steps for obtaining a second item image by cropping an image based on a sequence of contour lines to be compared include: acquiring the current contour line to be compared, comparing the current contour line to be compared with the previously cropped contour line for feature points; when a new feature point is detected and no new feature point disappears, continuing to acquire an image at the next location; when a new feature point is detected and a new feature point disappears simultaneously, triggering an image cropping condition; counting the number of new feature points appearing in each candidate item image when the cropping condition is met, selecting the candidate item image with the highest frequency of new feature point occurrences for cropping, and obtaining the second item image.
[0064] It should be noted that the image cropping condition is: the current contour line simultaneously has newly added feature points and feature points disappearing;
[0065] Specifically, multiple refresh points are evenly selected along the first contour line, and their initial coordinates are recorded as feature point references. These refresh points are used as feature reference points in subsequent contour line comparisons. When comparing the current contour line with the previous one, the current contour line is compared with the previously captured contour line. During this comparison, the contour line of the item to be sold changes as the item's position changes. When a new feature point appears on the contour line, the refresh points on the contour line are refreshed. This new feature point is obtained by comparing it with the image at the previous position, not from the previously captured image. Multiple images of the item to be sold at positions corresponding to newly appearing feature points are continuously captured as candidate item images. The system continues to detect whether the new feature point disappears when it appears. For example, when the image at the first position is the image of the first item, during the comparison between the image at the first position and the image at the second position, a new feature point appears in the image at the second position, but simultaneously, the disappearance of the new feature point may occur in the image at the first position. If feature points disappear from the image, since the feature points in the first image have already been recorded in the first item image, their disappearance does not affect the use of subsequent feature points. Therefore, there is no need to crop the image at the second position. Instead, the image at the third position is compared with the image at the first position. If new feature points appear and cover the new feature points at the second position, and if new feature points also disappear, then the image corresponding to the contour line at that position needs to be cropped. This ensures that the cropped image contains the most newly added feature points, so that each cropped image has common feature points while also having more different feature points. These are used for subsequent 3D modeling of the item for sale, which facilitates the subsequent identification of the item and improves the accuracy of identification. Through the dynamic contour line comparison mechanism, the precise capture of changes in the features of moving items is achieved. The dual judgment conditions of feature point appearance and disappearance ensure the accuracy of image cropping timing, effectively improving the completeness of item image acquisition and feature recognition.
[0066] For example, the initial image captured is an upright image of a beverage on a shelf (the first item image). The outline is extracted, and four refresh points are set. Five frames are captured along the motion trajectory. In the third frame, new feature points appear (e.g., label text becomes visible due to rotation), but there are no vanishing points, so capture continues. In the fourth frame, both new feature points (new features appear, such as a bottle cap that was previously invisible but becomes visible due to changes in the object's angle caused by the motion trajectory; in this case, the feature points on the bottle cap are considered new feature points) and feature point disappearance (partial occlusion of feature points is considered feature point disappearance) appear, triggering the cropping condition. The frequency of new feature points in each frame is counted; the fourth frame has the highest frequency and is cropped as the second item image. Finally, three images containing upright, side, and top views are stored for the recognition model. Through dual determination of feature point addition / disappearance, the timing of changes in the item's viewing angle is accurately captured, and the refresh point mechanism reduces the impact of lighting changes on outline extraction.
[0067] S2: Extract local feature information of the items for sale based on the item image set; construct a local item model corresponding to the items for sale based on the local feature information.
[0068] The steps for constructing a local item model corresponding to an item for sale based on local feature information include: extracting local feature points from each item image in the item image set and the relative positional relationship between the local feature points; obtaining the description of the feature points in two consecutively cropped images, performing feature point matching between adjacent images, identifying and extracting overlapping feature points; predicting the relative pose between adjacent images based on the overlapping feature points, recovering the three-dimensional feature points based on the overlapping feature points and the relative pose, aligning the three-dimensional feature points to a unified coordinate system, and generating a local item three-dimensional model.
[0069] It should be noted that overlapping feature points refer to feature points that appear in two consecutively cropped images. The process involves predicting the relative pose between adjacent images based on overlapping feature points, recovering 3D feature points based on overlapping feature points and relative poses, and aligning these 3D feature points to a unified coordinate system to generate a local 3D model of the object. The specific steps are as follows: extract feature points from adjacent images, find matching feature point pairs, and extract the feature points and descriptors. This can be achieved using feature detectors such as SIFT or ORB. Feature matching is performed using FLANN or BFMatcher, and RANSAC is applied to remove mismatches and retain inliers. Based on the matched feature points, the relative pose (rotation matrix and translation vector) between adjacent images is estimated. The pose can be obtained using essential matrix decomposition, and the essential matrix is calculated. The essential matrix decomposition yields four possible pose solutions. The correct solution is selected through triangulation verification. The feature points are projected onto a normalized camera coordinate system, and the 3D points are solved using linear triangulation. The coordinates of the 3D points can be recovered from the overlapping feature points of adjacent images, thus locally constructing a local 3D model of the object corresponding to the feature points in the image.
[0070] Specifically, during image cropping, it is necessary to ensure that each image contains overlapping feature points, meaning that there are overlapping local regions. The timing of the appearance and disappearance of feature points is controlled to ensure that adjacent regions overlap. Feature point descriptions are extracted for each item image, and feature points are matched between adjacent images. The relative pose between adjacent images is estimated based on the matched points. Using the matched points and camera pose, 3D points are recovered through triangulation and transformed to a unified coordinate system. Triangulation is used to generate local 3D models of items. Through local feature modeling technology, efficient and accurate identification of items in the vending machine is achieved.
[0071] S3: Collect reference image information of multiple reference items in the vending machine, and construct reference image models corresponding to the reference items based on the reference image information.
[0072] The steps of collecting reference image information of multiple reference items in the vending machine and constructing reference image models corresponding to the reference items based on the reference image information include: acquiring reference image information of each reference item in the vending machine at different placement angles; sequentially inputting the reference image information of each reference item at different placement angles into the initial image model to iteratively train the initial image model to obtain the reference image model corresponding to each reference item.
[0073] Specifically, the process involves acquiring reference image information of each reference item within the vending machine from different placement angles. This can be achieved by installing multiple cameras inside the vending machine or by taking pictures of the products inside when the vending machine doors open and close. The acquired reference image information is then preprocessed, including noise reduction and contrast enhancement, to improve image quality. An image model is initialized, which can be built based on a deep learning framework (such as a Convolutional Neural Network, CNN), serving as the initial image model. The reference image information of each reference item from different placement angles is then sequentially input into the initial image model, and the model is optimized through iterative training. During training, an appropriate loss function (such as Mean Squared Error (MSE) or Euclidean Distance) is used to evaluate the difference between the model's predictions and the true values, and the model parameters are adjusted accordingly. After multiple iterations of training, a reference image model corresponding to each reference item is obtained. This model can accurately identify and predict the image features of the reference items at different placement angles. Based on the obtained reference image models, and combined with 3D modeling techniques (such as stereo vision and structured light), a standard 3D model of each reference item within the vending machine is constructed. The constructed 3D model is optimized, including surface smoothing and noise removal, to improve model quality. By collecting standard reference information of reference items in the vending machine, a high-precision 3D model is constructed and feature points are labeled, which effectively improves the item recognition accuracy of the smart vending machine.
[0074] S4: Compare the local item model with multiple reference image models, extract the reference image model that meets the preset conditions, and use the reference item corresponding to the reference image model as the target item corresponding to the item to be sold.
[0075] The steps of comparing a local item model with multiple reference image models, extracting reference image models that meet preset conditions, and using the reference items corresponding to the reference image models as the target items corresponding to the items to be sold include: identifying and marking item feature points in the local item model; identifying and marking reference feature points in the reference image models; aligning the item feature points with the reference feature points, and using the reference image model containing the reference feature points that meet the preset conditions as the reference image model corresponding to the item feature points.
[0076] The step of aligning item feature points with reference feature points and using the reference image model containing the reference feature point that satisfies a preset overlap condition as the reference image model corresponding to the item feature point includes: acquiring item feature points and reference feature points, wherein the item feature points include a first item feature point and a second item feature point, and the reference feature points include a first reference feature point corresponding to the first item feature point and a second reference feature point corresponding to the second item feature point; aligning the first item feature point with the first reference feature points in multiple reference image models respectively, calculating the first overlap degree corresponding to each reference image model; and extracting the reference image model that satisfies the preset first overlap condition. The reference image model corresponding to the first overlap degree of the degree threshold is used to obtain a set of pre-selected reference image models; it is determined whether the number of pre-selected reference image models in the set is 1; if it is 1, the pre-selected reference image model is used as the reference image model corresponding to the feature point of the item; if it is not 1, the second feature point of the item is compared with the second reference feature point of multiple pre-selected reference image models in the set, the second overlap degree corresponding to each pre-selected reference image model is calculated, and the pre-selected reference image model corresponding to the second overlap degree that satisfies the preset second overlap degree threshold is used as the reference image model corresponding to the feature point of the item.
[0077] It should be noted that the preset first overlap threshold and the preset second overlap threshold are preset based on the actual application scenario, the distribution characteristics of the item feature points, and the accuracy of the reference image model. The preset second overlap threshold can be higher than or equal to the preset first overlap threshold to ensure the accuracy of the final determined reference image model. Overlap includes, but is not limited to, Euclidean distance calculation based on feature point coordinates, feature vector similarity calculation, or matching degree prediction based on machine learning models. When calculating the overlap, it is necessary to ensure that the position information of each item feature point and the relative position information between feature points correspond to the position information of the reference feature points and the relative position information between reference feature points, and then calculate the overlap after the correspondence.
[0078] The specific steps for calculating the overlap degree for each reference image model are as follows: When calculating the overlap degree, the feature vector or distance needs to be normalized to ensure consistency in comparisons between different feature points or different images. Assume there are two object feature points... and and their corresponding reference feature points and Calculate the Euclidean distance between them and normalize it to the degree of overlap: , ,in, Represents the coordinates of the first item feature point, where It is the x-axis. It is the ordinate. This represents the coordinates of the feature point of the second item. It is the x-axis. It is the ordinate. Indicates the feature point of the first item The coordinates of the first reference feature point, where It is the x-axis. It is the ordinate. Indicates the feature point of the second item The coordinates of the corresponding second reference feature point, where It is the x-axis. It is the ordinate. This represents the Euclidean distance between the first item feature point and the corresponding first reference feature point. Represents the first item feature point Its corresponding reference feature point The degree of overlap between them. The degree of overlap is obtained by normalizing the Euclidean distance; This represents the maximum possible distance, which is a preset constant or a distance value determined based on the application scenario. It is used to convert the distance into a degree of overlap (a value between 0 and 1). The calculation method for the second degree of overlap is the same as that for the first degree of overlap, except that... and The coordinates;
[0079] Specifically, the first item feature point and the first reference feature point are both marked as corner points and edge points in the local item model and multiple reference image models, respectively; the second item feature point and the second reference feature point are both set on the texture lines in the corresponding local item model and multiple reference image models, respectively, to determine the texture direction; by prioritizing the selection of the position of the first item feature point from multiple reference image models, the computational load of the overall similarity or overlap rate can be reduced. When the number of local item models selected by the first overlap rate is greater than one, texture comparison is then performed; corner points and texture contour points are salient feature points on the item, which are easy to extract and match. Through two rounds of overlap rate calculation and threshold judgment, the accuracy of the finally determined reference image model is ensured, thereby improving the recognition efficiency of the items to be sold.
[0080] S5: Obtain the item information of the target item in the video stream and the price corresponding to the target item; settle the bill for the item to be sold based on the price and item information corresponding to the target item.
[0081] The steps for obtaining item information and corresponding prices of target items in a video stream, and settling accounts for the items to be sold based on the prices and item information of the target items, include: obtaining information on the types of target items appearing in the video stream and identifying the quantity of each type of target item; counting the specific quantity of each type of target item and using the quantity of the target items and the total number of target item types as the item information of the target items; obtaining the corresponding price information for each identified target item; calculating the total price of the same type of target items based on the obtained price information; counting all types of target items and combining the total price of each type of target item to complete the settlement of the items to be sold.
[0082] Specifically, in the video stream, the user will purchase various types of target items. The unit price of each target item is obtained, and a tracking algorithm ensures continuous and accurate counting of the quantity of each type of target item in the video stream. Multiplying the quantity by the unit price yields the total price of one type of target item. Combining the total prices of multiple types of target items gives the total price of the items purchased by the user for sale. A settlement list containing the quantity, unit price, total price, and final settlement amount for each type of target item is generated. Based on the settlement list, the settlement process for the items for sale is completed. The system also retrieves the unit price for each target item by querying a pre-set price database; and / or, obtains price information for target items in real time via an external API interface; ensures the accuracy and real-time nature of price information to support accurate settlement; generates a settlement list containing the quantity, unit price, total price, and final settlement amount for each type of target item; provides multiple payment methods for users to complete payments according to the settlement list; updates inventory information, reduces the quantity of sold items, and records settlement information for subsequent querying and auditing.
[0083] This invention extracts at least two images from different angles from a video, extracts all different features from each image, fuses feature points based on identical feature points in adjacent images (including the relative positions of feature points), and constructs a local 3D product model. This local product model is compared with a reference model to identify the reference item corresponding to the reference image model that falls below preset conditions as the target item, and then the corresponding item price is determined for settlement. Through a dynamic contour line comparison mechanism, video frames meeting preset conditions are dynamically extracted to form multi-view product images, accurately capturing changes in the features of moving items, and using a dual judgment criterion of feature point appearance and disappearance. This approach ensures the accuracy of image capture timing, effectively improving the completeness and feature recognition of item images. By capturing multiple item images from different angles in the video stream, a local item model is constructed. This local item model is then compared with multiple reference image models to extract reference image models whose similarity meets preset conditions. The reference items corresponding to these reference image models are then used as the target items for sale. Through multi-model comparison, items in the vending machine can be accurately identified. The automated identification and settlement process significantly improves the settlement efficiency of the vending machine. AI technology enables automatic identification and rapid settlement of goods, enhancing the shopping experience and operational efficiency.
[0084] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0085] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A payment method for AI-based item recognition in vending machines, characterized in that, Includes the following steps: The system acquires a video stream of a user purchasing items from a vending machine, and extracts multiple images of the items for sale from the video stream to obtain an item image set, wherein the item image set includes at least two images of the items from different angles. Image acquisition points are set for each vending machine. A video stream of the entire process of a user purchasing items from the vending machine is captured based on these acquisition points, obtaining a continuous image sequence containing the movement information of the items for sale. The movement trajectory of the items for sale is analyzed from the video stream, and the initial position of the trajectory is determined. An image of the item for sale is acquired at the initial position, denoted as the first item image. Feature points are calibrated on the items for sale in the first item image to generate an initial feature point set. Contour lines of the items for sale are sequentially acquired at multiple positions along the movement trajectory, forming a dynamic contour line sequence. The currently acquired contour line is compared with the contour line at the previous position using feature point matching, and the following determination is performed: the current contour line to be compared is obtained, and the current contour line to be compared is... The feature points of the line are compared with those of the previously captured contour line. When a new feature point is detected and no new feature point disappears, the image at the next position is captured. When a new feature point is detected and a new feature point disappears, the image capture condition is triggered. The number of new feature points appearing in each candidate item image when the capture condition is met is counted, and the candidate item image with the highest frequency of new feature points is selected for capture to obtain the second item image. The contour line acquisition and comparison steps are repeated until the image capture of the entire motion trajectory is completed, and second item images from multiple perspectives are obtained. The first item image and multiple second item images are associated and stored to form an image set containing at least two item images from different angles. Local feature information of items for sale is extracted from the item image set; local item models corresponding to the items for sale are constructed based on the local feature information. Collect reference image information of multiple reference items in the vending machine, and construct reference image models corresponding to the reference items based on the reference image information; The local item model is compared with multiple reference image models, and the reference image model that meets the preset conditions is extracted. The reference item corresponding to the reference image model is used as the target item corresponding to the item to be sold. Obtain the item information and corresponding price of the target item in the video stream; settle the bill for the item to be sold based on the price and item information of the target item.
2. The settlement method for AI recognition of vending machine items according to claim 1, characterized in that: The step of extracting video frames that meet preset conditions from the video stream and capturing the corresponding images of the items for sale, denoted as the second item image, includes: Extract the outline of the item to be sold from the first item image to obtain the initial outline, which is denoted as the first outline; Multiple refresh points are set based on the first contour line; Images from multiple locations are sequentially acquired along the movement trajectory of the items to be sold, resulting in a sequence of images of the items to be selected. Extract the outline of the item to be sold corresponding to each candidate item image to form a sequence of outlines to be compared; based on the sequence of outlines to be compared, perform image cropping to obtain the second item image.
3. The settlement method for AI recognition of vending machine items according to claim 1, characterized in that: The steps for constructing a local item model corresponding to the item for sale based on local feature information include: Extract local feature points from each item image in the item image set and the relative positional relationships between these local feature points; Obtain descriptions of feature points in two consecutively cropped images, perform feature point matching between adjacent images, and identify and extract overlapping feature points; The relative pose between adjacent images is predicted based on overlapping feature points. Three-dimensional feature points are recovered based on overlapping feature points and relative pose, and the three-dimensional feature points are aligned to a unified coordinate system to generate a local three-dimensional model of the object.
4. The settlement method for AI recognition of vending machine items according to claim 1, characterized in that: The steps of collecting reference image information of multiple reference items in the vending machine and constructing reference image models corresponding to the reference items based on the reference image information include: Obtain reference image information of each reference item in the vending machine at different placement angles; The reference image information of each reference object at different placement angles is sequentially input into the initial image model to gradually train the initial image model iteratively, so as to obtain the reference image model corresponding to each reference object.
5. The settlement method for AI recognition of vending machine items according to claim 1, characterized in that: The step of comparing a local item model with multiple reference image models, extracting reference image models that meet preset conditions, and using the reference item corresponding to the reference image model as the target item corresponding to the item to be sold includes: Identify and label the feature points of the object in the local object model, and identify and label the reference feature points in the reference image model; The object feature points are superimposed with reference feature points, and the reference image model containing the reference feature points that meet the preset conditions of superposition is used as the reference image model corresponding to the object feature points.
6. The settlement method for AI recognition of vending machine items according to claim 5, characterized in that: The step of aligning the object feature points with reference feature points and using the reference image model containing the reference feature points that meet preset conditions as the reference image model corresponding to the object feature points includes: Obtain item feature points and reference feature points, wherein the item feature points include a first item feature point and a second item feature point, and the reference feature points include a first reference feature point corresponding to the first item feature point and a second reference feature point corresponding to the second item feature point; The first item feature point is overlapped with the first reference feature point in multiple reference image models, and the first overlap degree corresponding to each reference image model is calculated. Extract the reference image model corresponding to the first degree of overlap that meets the preset first degree of overlap threshold, and obtain a pre-selected reference image model set; Determine whether the number of preselected reference image models in the preselected reference image model set is 1; If the value is 1, then the pre-selected reference image model is used as the reference image model corresponding to the feature points of the item; If it is not 1, then the second item feature point is compared with the second reference feature point of multiple pre-selected reference image models in the pre-selected reference image model set, the second overlap degree corresponding to each pre-selected reference image model is calculated, and the pre-selected reference image model corresponding to the second overlap degree that meets the preset second overlap degree threshold is used as the reference image model corresponding to the item feature point.
7. The settlement method for AI recognition of vending machine items according to claim 1, characterized in that: The process involves acquiring the item information of the target item in the video stream and the price corresponding to the target item; The steps for settling accounts for items to be sold based on the price and item information corresponding to the target item include: Obtain information on the types of target items appearing in the video stream and identify the quantity of each type of target item; The specific quantity of each type of target item is counted, and the quantity of the target items and the total number of target item types are used as the item information of the target items; For each identified target item, obtain its corresponding price information; based on the obtained price information, calculate the total price of similar target items; The system identifies all types of target items and aggregates the total price of each type to complete the settlement of the items for sale.