Binocular Pedestrian Detection Using Dual-Stream Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current pedestrian detection methods, such as monocular video detection and binocular vision systems, face challenges in accurately detecting pedestrians, especially in scenarios with shadows, footprints, or stationary individuals, and require expensive 3D data acquisition systems like LiDAR, making them impractical for common public places.
Innovation Solution
A binocular pedestrian detection system utilizing a dual-stream deep learning neural network that combines binocular images and disparity maps to enhance detection accuracy, featuring a binocular camera setup with image/video processing ASIC and a controller, which captures and processes images to calculate disparity maps and train a dual-stream neural network for robust pedestrian detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If monocular RGB image characteristics are used for pedestrian detection, then the system is simple and low-cost, but detection accuracy is poor for difficult targets such as shadows, footprints, or pedestrians wearing hats
Solution Approach 1:
The patent combines monocular RGB image detection with binocular disparity map detection into a unified detection framework. The system merges features from both monocular and binocular streams, using the strengths of each approach to compensate for their individual weaknesses, thereby improving overall detection accuracy without requiring a complete system overhaul
Solution Approach 2:
The patent uses composite feature representation by combining RGB image features with disparity map features. This composite feature set leverages both color/texture information from RGB images and depth/distance information from disparity maps, creating a more robust detection signature that improves accuracy for difficult targets
2Reliability
If binocular disparity maps are used to select regions of interest, then false pedestrian targets such as shadows and footprints can be removed, but detection accuracy for special pedestrian targets such as those wearing hats remains poor
Solution Approach 1:
The patent creates composite detection features by combining RGB image characteristics with disparity map characteristics. This composite approach allows the system to maintain the false target rejection capability of disparity-based methods while adding RGB feature information that helps identify special pedestrian targets such as those wearing hats
Solution Approach 2:
The patent segments the detection process into multiple independent streams: one stream processes RGB image features while another processes disparity map features. Each stream can be optimized independently, and their results are combined to achieve both false target rejection and accurate pedestrian detection
3Measurement precision
If 3D data acquisition systems such as LiDAR are used, then three-dimensional locations can be accurately determined, but the system becomes expensive and difficult to equip in common public places
Solution Approach 1:
The patent creates a virtual 3D representation by computing disparity maps from binocular stereo images. This computational approach generates depth information that mimics the functionality of LiDAR systems without requiring expensive 3D sensing hardware, effectively copying the depth perception capability through software processing of standard camera images
Solution Approach 2:
The patent replaces the mechanical/optical 3D sensing system (LiDAR) with a computational vision system using binocular stereo cameras and disparity map algorithms. This substitution achieves similar 3D location accuracy through image processing and mathematical computation rather than active laser ranging
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of present disclosure relates to a binocular pedestrian detection system (BPDS). BPDS includes: a binocular camera to capture certain binocular images of pedestrians passing through a predetermined area, an image/video processing ASIC to process binocular images captured, and a binocular pedestrian detection system controller having a processor, a network interface, and a memory storing computer executable instructions. When executed by processor, computer executable instructions cause processor to perform: capturing, by binocular camera, binocular images of pedestrians, binocularly rectifying binocular images, calculating disparity maps of binocular images rectified, training a dual-stream deep learning neural network, and detecting pedestrians passing through predetermined area using dual-stream deep learning neural network trained. Dual-stream deep learning neural network includes a neural network for extracting disparity features from disparity maps of binocular images, and a neural network for learning and fusing features from rectified left images and disparity maps of binocular images.