Light-Field Object Tracking via Multi-View Photo-Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional object tracking methods face challenges in maintaining accuracy and reliability due to changes in object appearance over time, caused by variations in pose, camera viewpoint, and lighting conditions, which can lead to tracker drift and inclusion of background information in the model, especially with initial specification inaccuracies and partial occlusions.
Innovation Solution
The method employs a light-field camera to capture multiple viewpoints, ensuring photo-consistency across views to discriminate object pixels from background pixels, rejecting features not identifiable in all views and refining the appearance model online to enhance tracking accuracy and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional object tracking methods are used with single-view cameras, then the tracking process is simple and fast, but the tracking accuracy deteriorates due to inability to discriminate object pixels from background pixels under varying appearance conditions
Solution Approach 1:
The patent transitions from single-view (2D) tracking to multi-view (3D spatial + temporal) tracking by capturing light-field data from multiple camera viewpoints. This dimensional expansion enables discrimination of object pixels from background pixels through photo-consistency checks across different views, resolving the accuracy-complexity contradiction by adding spatial dimensions rather than increasing algorithmic complexity alone.
Solution Approach 2:
The patent introduces light-field captures from multiple views as an intermediary between the object and the tracking system. These intermediate multi-view observations serve as mediators that enable photo-consistency verification, allowing the system to accurately distinguish object pixels from background pixels without requiring overly complex single-view processing.
2Adaptability or versatility
If the appearance model is updated online to adapt to appearance changes, then the tracker can handle pose and lighting variations, but background information is incorporated into the model causing tracker drift
Solution Approach 1:
The patent applies local quality by performing photo-consistency checks specifically for pixels within the bounding box region. Only pixels that satisfy photo-consistency constraints across multiple views are incorporated into the updated appearance model. This localized verification ensures that adaptability is achieved while preventing background contamination, as only genuine object pixels passing the consistency test are included in the model update.
Solution Approach 2:
The patent implements feedback through photo-consistency verification that provides information about which pixels should be included in the appearance model update. This feedback mechanism allows the system to adapt to appearance changes while maintaining reliability by using consistency information across views to guide which pixels are incorporated, preventing tracker drift through selective model updating.
3Measurement precision
If multiple views from light-field camera are used to ensure photo-consistency, then object segmentation accuracy is improved, but the processing time and computational load increase
Solution Approach 1:
The patent applies segmentation by dividing the light-field data processing into distinct stages: initial object localization in one view, photo-consistency verification across multiple views for pixels within the bounding box, and selective model updating. This segmentation of the processing pipeline enables accurate object segmentation through multi-view consistency checks while managing computational load by processing only relevant pixels rather than entire images.
Solution Approach 2:
The patent implements partial action by performing photo-consistency checks only for pixels within the bounding box region rather than processing the entire image. This selective processing maintains high segmentation accuracy for the object of interest while significantly reducing computational time and processing load compared to full-image multi-view processing.
Data Source
Figure 1~2b
Figure 3
Figure 4
AI summary
A method and an electronic device (20, 40) for object tracking in a sequence of light-field captures are described. A data acquisition unit (22) acquires (S0) a sequence of light-field captures, wherein each light-field capture comprises a plurality of views. A feature determining unit (23) determines (S1) features of a visual appearance model for an object of interest in a reference view of a first light-field capture. A feature matching unit (24) matches (S2) the features in the reference view and in the further views of the first light-field capture. A feature discarding unit (25) discards (S4) features that cannot be well matched in all views of the first light-field capture. An appearance model building unit (26) builds (S7) a visual appearance model for the object of interest based on the remaining features. Finally, a movement tracking unit (27) tracks (S8) the movement of the object of interest in the sequence of light-field captures using the visual appearance model.