Stereo Image Depth Fusion for Shopper Tracking Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing in-store analytics systems face challenges in accurately detecting, tracking, and counting shoppers using video frames, particularly in distinguishing between shoppers and staff, and handling variations in shopper postures and occlusions.
Innovation Solution
The system employs a method that fuses image and depth modalities by obtaining stereo image pairs, rectifying frames, computing stereo disparity, and using machine learning models to detect and track objects, creating robust object tracks through a hypergraph optimization framework.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional single-modality object detection is used in video frames, then the system complexity is low, but the detection accuracy and robustness deteriorate due to missed detections and false positives
Solution Approach 1:
The patent combines multiple detection modalities (depth modality and image modality) into a unified detection system. The depth information from stereo cameras and the visual information from standard cameras are merged to create complementary detection results, where depth data provides robustness against occlusions and posture variations while image data provides detailed visual characteristics, thereby improving overall detection accuracy without excessive complexity increase
Solution Approach 2:
The patent introduces depth information as an additional dimension beyond traditional 2D image analysis. By incorporating the third dimension (depth/distance) through stereo disparity computation, the system gains enhanced ability to distinguish objects in 3D space, separate occluded objects, and detect posture variations, significantly improving detection precision in complex retail environments
2Reliability
If multiple detection modalities are fused to improve accuracy, then detection robustness improves, but computational complexity and processing time increase
Solution Approach 1:
The patent performs preliminary rectification and disparity computation on stereo image pairs before the main detection process. By pre-processing the depth information and creating rectified stereo pairs in advance, the system reduces the computational burden during real-time detection, allowing robust multi-modality fusion without excessive processing delays during critical detection phases
Solution Approach 2:
The detection process is segmented into distinct modules: depth modality detection, image modality detection, and result fusion. Each module processes information independently and contributes to the final detection result, allowing parallel processing and optimizing the balance between robustness and processing efficiency by selectively applying computation where most beneficial
3Measurement precision
If stereo disparity computation is performed on all frames, then depth accuracy improves, but computational load and processing speed deteriorate
Solution Approach 1:
The patent applies stereo disparity computation selectively rather than uniformly to all video frames. By performing depth computation only when necessary (e.g., when object detection confidence is low, when occlusion is detected, or at specific frame intervals), the system maintains adequate depth accuracy for critical detections while reducing overall computational load and improving processing speed for the video stream as a whole
Data Source
AI summary
Various embodiments are disclosed for detecting, tracking and counting objects of interest in video frames using fusion of image and depth modalities. In an embodiment, a method comprises: obtaining multiple frames of stereo image pairs from an image capturing device; rectifying each frame of the stereo image pairs; computing a stereo disparity for each frame of the stereo image pairs; determining a first set of object detections in each frame using the computed stereo disparity; determining a second set of object detections in each left or right frame of the stereo image pair using one or more machine learning models; fusing the first and second sets of object detections; and creating tracks based on the fused object detections.