3D Point Cloud Item Recognition for Checkout Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing item recognition systems face challenges in accurately identifying items in checkout processes due to occlusion, complex product appearances, and the need for extensive training data, especially in environments with limited space for self-service terminals and frequent changes in product inventory.
Innovation Solution
A system that uses multiple cameras to capture images from different angles, combines depth information and RGB data to create 3D point clouds, and employs machine learning models for item segmentation and classification, allowing for improved accuracy and reduced training time through iterative learning and transfer learning techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complex convolutional neural networks are used for item recognition, then item recognition accuracy is improved, but the need for large volumes of training data increases
Solution Approach 1:
The patent segments the item recognition process into multiple stages: initial detection using simplified models, followed by refined classification using more complex models only for detected candidates. This segmentation allows accurate recognition without requiring all complex models to be trained on extensive data sets.
Solution Approach 2:
The patent introduces depth information as an additional dimension to traditional 2D image recognition. By incorporating 3D point cloud data and depth maps, the system creates multi-dimensional feature representations that improve recognition accuracy while reducing dependency on large training data volumes.
2Measurement precision
If multiple cameras are used to capture items from different angles, then item recognition accuracy is improved, but device complexity increases
Solution Approach 1:
The patent merges data from multiple cameras into unified 3D point clouds and depth maps. By combining information from multiple sources into integrated data structures, the system achieves improved recognition accuracy while managing complexity through data fusion rather than processing multiple independent streams.
Solution Approach 2:
The patent introduces 3D point clouds and depth maps as intermediary representations between raw camera images and final item classification. These intermediaries simplify the processing pipeline by providing structured geometric information that bridges the gap between multi-camera input and recognition output.
3Device complexity
If traditional 2D image recognition is used, then device complexity is kept low, but recognition accuracy deteriorates due to occlusion
Solution Approach 1:
The patent transitions from 2D image processing to 3D point cloud and depth map analysis. This dimensional upgrade allows the system to see through occlusions by utilizing depth information and spatial relationships, achieving accurate recognition even when items are partially hidden.
Solution Approach 2:
The patent performs preliminary 3D reconstruction and depth analysis before final item classification. By pre-processing images into 3D point clouds and identifying occlusion patterns early in the pipeline, the system prepares enhanced feature representations that improve subsequent recognition accuracy without adding significant complexity.
4Productivity
If self-service terminals are installed for automated checkout, then productivity is improved, but space requirements increase
Solution Approach 1:
The patent designs the automated recognition system to perform multiple functions: item detection, classification, occlusion handling, and transaction initiation. This multi-functionality consolidates what would traditionally require separate terminal components into a unified system that achieves high productivity in compact form.
Solution Approach 2:
The patent implements a self-service system where customers place items on a mat with embedded cameras, and the system automatically performs recognition and charging without human intervention. This eliminates the need for traditional self-service terminals with displays and interfaces, achieving automated checkout in a space-efficient manner.
Data Source
AI summary
A first Machine-Learning Model (MLM) is processed on multiple images of a scene, each scene comprising a different perspective view of each of a plurality of items. The first MLM produces masks for the items within each image, each mask representing a portion of a given item within a given image. Depth information associated with the images and the masks are processed to isolate each portion of each item within each image. A single scene image is generated from the images by stitching each image’s pixel data from each portion or each image into a composite item image within the single scene image. Each item’s composite item image is passed to a second MLM and the second MLM returns an item code for the corresponding item associated with the corresponding composite item image. The item codes for each item is passed to a transaction manager to process a transaction.