Multiview Item Recognition Using 3D Depth Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Item recognition in checkout systems is challenging due to occlusion issues, where items block each other's view, making it difficult for cameras to capture clear images for accurate identification, especially in multi-item scenarios, and this problem is exacerbated in convenience stores lacking self-service terminals.
Innovation Solution
A system that uses multiple cameras capturing images from different angles to create a multiview association of items, employing Convolutional Neural Networks (CNNs) and depth information to map pixel coordinates to real-world locations, allowing for accurate item identification even when items are partially occluded, by generating bounding boxes and 3D point clouds to isolate and recognize items.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple items are placed in one area for checkout, then transaction efficiency is improved, but item occlusion increases making recognition difficult
Solution Approach 1:
The patent transitions from 2D image processing to 3D spatial understanding by implementing depth estimation and 3D point cloud generation. This allows the system to perceive items in three-dimensional space, distinguish occluded items from visible ones, and reconstruct item positions even when partially blocked, thereby maintaining high transaction efficiency while solving the occlusion problem.
Solution Approach 2:
The patent introduces depth information and 3D point clouds as intermediary representations between the camera images and item recognition. These intermediaries bridge the gap between 2D image data and 3D item positions, enabling the system to infer item locations and identities even when directly visible in images is limited due to occlusion.
2Productivity
If frictionless checkout is implemented without barcode scanning, then checkout speed is improved, but item identification accuracy decreases due to occlusion
Solution Approach 1:
The system uses 3D depth information and point cloud processing to compensate for the lack of barcode scanning. By reconstructing item positions and orientations in 3D space, the system can identify items based on their spatial characteristics and visual features even when partially occluded, maintaining identification accuracy while enabling frictionless checkout.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously refines item identification by comparing detected features with known item databases, using depth information and multiple view synthesis to verify and correct identifications, thereby maintaining high accuracy without manual barcode verification.
3Reliability
If depth information and multiview association are used to improve item recognition, then occlusion handling is improved, but system complexity increases
Solution Approach 1:
The patent implements a unified processing framework that handles multiple functions (depth estimation, 3D point cloud generation, multiview synthesis, and item recognition) within a single integrated system. This multi-functional approach reduces overall system complexity compared to having separate specialized systems for each function, while maintaining high recognition reliability through comprehensive multiview association.
Data Source
AI summary
Images of a transaction area comprising items are received during a checkout. Red-Green-Blue (RGB) data and Depth data are received with each image. Pixels captured in any given image by a given camera are pre-aligned with physical locations of a transaction area. Depth data provided by each camera and that camera's pre-alignment to the area are processed to map pixels in each image taken to X-Y coordinates within the area. X-Y coordinates for each item and for each image are grouped together as a single item within the area. RGB data for each image and item is used as a set of image patches per item. For each item, the corresponding patches are passed to a classification Machine-Learning Model (MLM) that returns an item code for each patch. A particular item code is selected for each set and the item codes are used to process the checkout.