Self-Checkout Commodity Image Matching for Error Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional self-checkout systems face challenges in efficiently detecting fraudulent or erroneous operations, such as inaccurate scanning or barcode manipulation, due to the need for extensive image registration and complex customer behavior monitoring, which are impractical and vulnerable to malicious actions.
Innovation Solution
An erroneous operation prevention system using a learned multimodal foundation model, like CLIP, to calculate the similarity between extracted commodity images and registered data, notifying clerks of discrepancies when the similarity falls below a threshold, and optionally utilizing a vision-language fusion model for more accurate detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image recognition technique is used to automatically register commodities, then checkout accuracy is improved, but the system complexity and time required for image registration increase significantly
Solution Approach 1:
The patent extracts only the necessary visual features from commodity images (such as packaging color, shape, and distinctive markings) rather than processing complete images. This selective feature extraction reduces the complexity of image registration while maintaining sufficient accuracy for commodity identification during checkout.
Solution Approach 2:
The system divides the commodity identification task into separate modules: barcode scanning for primary identification, visual feature extraction for verification, and pattern matching for confirmation. This segmentation allows each module to process information independently, reducing overall system complexity while improving robustness.
2Measurement precision
If all commodity images are registered in advance, then commodity identification accuracy is improved, but the time and man-hours required for image registration increase significantly
Solution Approach 1:
The system performs preliminary extraction and storage of key visual features from commodity images during the registration phase, but does not require complete image processing for all commodities in advance. Only essential feature data is pre-processed and stored, reducing the time investment while maintaining identification accuracy during actual checkout operations.
Solution Approach 2:
Instead of processing all possible commodity images completely, the system processes only the necessary visual features partially. This partial processing approach reduces the total time required for image registration while providing sufficient accuracy for the checkout function.
3Reliability
If customer behavior is monitored through image capture, then erroneous operation detection is improved, but customer awareness of monitoring increases vulnerability to malicious actions
Solution Approach 1:
The system uses visual feature extraction as an intermediary between image capture and commodity identification. Instead of directly analyzing complete customer behavior images, the system extracts specific visual features from commodities and compares them against registered data. This intermediary approach maintains detection capability while reducing the likelihood of customer awareness and malicious manipulation.
4Productivity
If barcode scanning is used for commodity registration, then checkout speed is improved, but fraudulent acts such as barcode replacement are not prevented
Solution Approach 1:
The patent combines barcode scanning with visual feature verification into a unified registration system. The barcode provides quick identification, while the visual feature extraction and pattern matching provide fraud detection. Both methods work together to verify commodity identity, maintaining checkout speed while preventing barcode replacement fraud.
Solution Approach 2:
The system provides feedback by comparing extracted visual features against registered commodity data and notifying users of any mismatches. This feedback mechanism allows the system to detect fraudulent barcode replacements while maintaining the speed of barcode scanning, as the visual verification occurs in parallel or provides immediate correction opportunities.
Data Source
AI summary
A camera captures an image of a checkout state of a customer at a self-checkout machine, and transmits the captured image to a management device. The management device extracts a commodity image from the received image. When commodities have been registered by the customer scanning the commodities, the self-checkout machine transmits data of checkout commodities having been registered, to the management device. Using the extracted commodity image and commodity names in the checkout commodity data, the management device calculates a similarity therebetween by using a learned multimodal foundation model. If the calculated similarity is less than a threshold value, a clerk is notified of an erroneous operation warning through a clerk terminal.