Dynamic Occlusion Sensitivity Map for 3D-CNN Video Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional occlusion sensitivity maps (OSM) are not applicable to 3D-CNNs for moving image recognition, as they fail to account for the movement of occluded areas across frames, making it difficult to visualize important areas for class identification in moving images.
Innovation Solution
A method that calculates optical flow to track the movement of areas in a movie and generates occluded movie data by adjusting the occluded area in each frame based on this flow, allowing for comparison of class likelihoods before and after occlusion to identify crucial areas for class identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional occlusion sensitivity map (OSM) technology is applied to 3D-CNN for moving image recognition, then the visualization process is simple, but the visualization accuracy is poor because the occluded area does not follow the movement of the area corresponding to the occluded area
Solution Approach 1:
The patent applies the dynamics principle by making the occluded area dynamic instead of static. The occluded area is moved according to the optical flow of the corresponding area in the movie data, allowing it to follow the movement of objects across frames. This dynamic adjustment resolves the contradiction by introducing motion tracking complexity that significantly improves visualization accuracy for moving image recognition.
Solution Approach 2:
The patent introduces optical flow as an intermediary element that mediates between the occluded area and the corresponding area in the movie data. The optical flow calculates pixel movement to determine how the occluded area should be positioned in each frame, serving as a bridge that enables accurate tracking of moving areas without requiring direct complex pixel-by-pixel comparison.
2Ease of operation
If the occluded area is set at the same position in each frame, then the processing is simple, but the visualization result cannot be understood by users because it does not follow the movement of the area
Solution Approach 1:
The patent transforms the static occluded area into a dynamic one that moves with the content of the movie. By calculating optical flow and applying it to the occluded area position, the system maintains processing simplicity through automated tracking while dramatically improving interpretability, as users can now see the occluded area following the movement of objects throughout the sequence.
Solution Approach 2:
The system uses optical flow calculation as a feedback mechanism that continuously monitors pixel movement and adjusts the occluded area position accordingly. This feedback loop ensures that the occluded area remains aligned with the corresponding area in each frame, making the visualization results interpretable and meaningful for understanding 3D-CNN behavior in moving image recognition.
3Ease of manufacture
If conventional OSM is applied to movie data, then the method is easy to implement, but it is difficult to obtain meaningful visualization results because the occluded area does not track movement
Solution Approach 1:
The patent preserves movement information by making the occluded area dynamic and responsive to optical flow. This approach maintains ease of implementation through the use of standard optical flow algorithms while preventing information loss about object movement, allowing the visualization to accurately reflect which areas are important for class identification throughout the movie sequence.
Solution Approach 2:
Optical flow serves as an intermediary that captures and transmits movement information from the movie data to the occluded area positioning. This intermediary mechanism preserves essential motion information without complicating the overall implementation, as the optical flow calculation is a well-established technique that can be integrated into the existing OSM framework.
Data Source
AI summary
A non-transitory computer-readable recording medium has stored therein a program that causes a computer to execute a process, the process including acquiring movie data including a plurality of consecutive frames calculating first likelihood of a class of the movie data by inputting the acquired movie data to a trained model, calculating an optical flow indicating movement of an area included in the movie data, based on the movie data generating occluded movie data by setting an occluded area in each of the frames included in the movie data, based on the optical flow, calculating second likelihood of a class of the occluded movie data by inputting the occluded movie data to the model identifying an area that affects identification of the class among areas in the movie data, based on the first likelihood and the second likelihood and displaying the identified area that affects identification of the class.