Self-Checkout Error Detection Using Multimodal Image Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing self-checkout systems face challenges in efficiently detecting fraudulent or erroneous operations, such as inaccurate scanning or barcode manipulation, due to the complexity and practical limitations of image registration and customer behavior management, which can be exploited by malicious customers.
Innovation Solution
An erroneous operation prevention system utilizing a learned multimodal foundation model, like CLIP, to calculate the similarity between extracted commodity images and registered data, and notify clerks of potential errors when the similarity falls below a threshold, along with optional enhancements using vision-language fusion models for more accurate detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If image recognition technique is used to automatically register commodities, then fraudulent acts and erroneous operations can be prevented, but a huge amount of time and man-hours are required for registering the images of all commodities
Solution Approach 1:
The patent applies preliminary action by pre-registering reference images of commodities in a database before the self-checkout process begins. This allows the image recognition system to immediately use the pre-registered images for comparison during customer checkout, eliminating the need for real-time image registration and reducing the time required for fraudulent act prevention.
Solution Approach 2:
The patent uses copying by creating reference images (copies) of commodities and storing them in a database. During checkout, the system compares images of actual commodities with these pre-existing reference copies to identify mismatches, enabling fraudulent act detection without requiring real-time registration of all commodity images.
2Measurement precision
If all commodities are registered with reference images, then accurate commodity identification can be achieved, but each commodity needs to be re-registered if the package is changed
Solution Approach 1:
The patent applies dynamics by making the reference image database adaptable to package changes. When a commodity package is changed, the system can update the corresponding reference image in the database, allowing the identification accuracy to be maintained without requiring complete re-registration of all commodities. This dynamic update mechanism reduces the complexity of maintaining accurate commodity identification.
3Measurement precision
If customer behavior is captured and managed with in-store cameras, then commodity registration accuracy can be improved, but complicated management and handling of customer and commodity data is required
Solution Approach 1:
The patent applies the extraction principle by focusing only on extracting and comparing images of commodities during the checkout process, rather than capturing and managing comprehensive customer behavior data. The system extracts relevant visual information (commodity images) from the checkout scene and compares it with reference images, significantly reducing the complexity of data management while maintaining registration accuracy.
4Reliability
If image recognition technique is used to identify commodities, then fraudulent acts can be detected, but malicious customers may hide commodities from image capturing means to prevent registration
Solution Approach 1:
The patent applies feedback by implementing a verification mechanism where the system compares captured commodity images with pre-registered reference images and provides immediate feedback to the customer. If a mismatch is detected (indicating potential fraudulent activity or package change), the system notifies the customer, creating a feedback loop that helps detect and prevent fraudulent acts while maintaining customer trust.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A camera 10 captures an image of a checkout state of a customer at a self-checkout machine 20, and transmits the captured image to a management device 40 (S1). The management device 40 extracts a commodity image from the received image (S2). When commodities have been registered by the customer scanning the commodities, the self-checkout machine 20 transmits data of checkout commodities having been registered, to the management device 40 (S3). Using the extracted commodity image and commodity names in the checkout commodity data, the management device 40 calculates a similarity therebetween by using a learned multimodal foundation model (S4). If the calculated similarity is less than a threshold value, a clerk is notified of an erroneous operation warning through a clerk terminal 50 (S5).