Joint Bilateral Filter Depth Map Estimation on Integrated Circuit
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating depth maps using joint bilateral filters are computationally complex, making them unsuitable for cost-efficient implementation in consumer devices like integrated circuits.
Innovation Solution
The integrated circuit performs the joint bilateral filter at a coarser resolution by conducting slicing operations on down-sampled volumes, reducing the number of division operations and allowing the slicing operation to be performed in software, while splatting and interpolation operations are handled in hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a joint bilateral filter is implemented using conventional methods, then depth map estimation accuracy is improved, but computational complexity increases significantly
Solution Approach 1:
The patent divides the computational process into distinct hardware and software components. The hardware circuit performs splatting and interpolation operations on down-sampled volumes, while the software performs slicing operations. This segmentation allows complex joint bilateral filtering to be broken into manageable parts that can be efficiently distributed between hardware and software execution environments.
Solution Approach 2:
The patent introduces a resolution dimension by performing operations at multiple scales. Instead of processing full-resolution images throughout, the system down-samples volumes for certain operations and interpolates results back. This dimensional approach to resolution management significantly reduces the number of division operations required while maintaining depth map estimation quality.
2Productivity
If the joint bilateral filter is approximated using conventional methods, then computational burden is reduced, but implementation cost remains high for consumer devices
Solution Approach 1:
The patent applies different processing qualities to different parts of the computation. Critical operations that benefit from parallel execution and deterministic timing (splatting and interpolation) are implemented in hardware, while less time-critical operations (slicing) are performed in software. This local differentiation of implementation quality optimizes both speed and cost for consumer device applications.
Solution Approach 2:
The patent introduces down-sampled volumes as an intermediary representation between the input image and the final depth map. These intermediate volumes at reduced resolution serve as a computational bridge, allowing the system to perform complex filtering operations with fewer calculations while still producing high-quality results through subsequent interpolation.
3Measurement precision
If slicing operations are performed on full-resolution data, then depth map quality is maintained, but the number of division operations increases dramatically
Solution Approach 1:
The patent performs down-sampling of volumes before the slicing operation, preparing the data in advance at a reduced resolution. This preliminary action reduces the computational workload for the subsequent slicing and division operations. The full-resolution depth map quality is then restored through interpolation after the low-cost slicing operation completes.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
An integrated circuit and computer-implemented method are provided for estimating a depth map from an image using a joint bilateral filter at reduced computational complexity. For that purpose, image data of an image is accessed as well as depth data of a template depth map. A joint bilateral filter is then applied to the template depth map using the image data as a range term in the joint bilateral filter, thereby obtaining an image-adapted depth map as output. The applying of the joint bilateral filter comprises initializing a sum-of-weighted-depths volume and a sum-of-weights volume as respective empty data structures in a memory, performing a splatting operation to fill said volumes, performing a slicing operation to obtain an image-adapted depth volume, and performing an interpolation operation to obtain an image-adapted depth value of the image-adapted depth map for each pixel in the image.Compared to known methods for estimatinga depth map from an image using a joint bilateral filter, a reduced computational complexity is obtained.