3D to 2D GAN for Moving Object Removal in Video Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for removing moving objects from digital video frames face issues with accuracy, efficiency, and flexibility due to the reliance on optical flow techniques, which result in inaccurate object removal, high computational resource usage, and rigidity in handling arbitrary object shapes.
Innovation Solution
A 3D to 2D generative adversarial neural network is employed to leverage temporal information, using gated convolution layers and a self-attention mechanism to accurately and efficiently remove moving objects by borrowing information from past frames and generating realistic textures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If optical flow techniques are used to predict object movement, then object removal can be performed, but accuracy deteriorates when camera movement is present resulting in ghost contours and blurry lines
Solution Approach 1:
The patent replaces the optical flow mechanical system with a deep learning-based system. Instead of using traditional optical flow algorithms that rely on pixel intensity comparisons and motion vectors, the invention employs convolutional neural networks to directly predict object positions and movements. This substitution eliminates the fundamental limitations of optical flow in handling camera movement, as the neural network learns to distinguish between object motion and camera motion through training on video data, thereby maintaining high accuracy without producing ghost contours or blurry artifacts
Solution Approach 2:
The patent changes the parameters used for motion prediction from optical flow-based pixel displacement to deep learning-based feature extraction. By transforming the input representation from raw pixel intensities to learned feature embeddings, and changing the output from motion vectors to direct position predictions, the system achieves robustness against camera movement. The neural network learns to invariantly represent objects across different camera positions and orientations, maintaining precise object localization even when the camera moves
2Reliability
If standard convolutional neural networks are used to process high-dimension digital images, then spatial relationships are preserved, but computational resource usage increases excessively requiring vast processing cycles and memory
Solution Approach 1:
The patent segments the convolutional processing into hierarchical stages with increasing abstraction. Instead of applying full-resolution convolutions throughout the network, the invention divides processing into early stages that handle local spatial relationships at lower resolutions and later stages that refine global spatial relationships. This segmentation allows the network to preserve spatial relationships where needed while reducing computational load through progressive downsampling and selective feature processing at different hierarchical levels
Solution Approach 2:
The patent introduces temporal dimensionality to the processing architecture by leveraging video frame sequences. Instead of processing each high-dimension image frame independently with computationally expensive convolutions, the invention uses temporal information from adjacent frames to predict and fill in missing data, effectively reducing the computational burden on individual frames. This dimensional transformation allows the system to maintain spatial relationship preservation while dramatically reducing per-frame computational requirements through temporal coherence assumptions
3Ease of manufacture
If content encoders with fixed shapes are used, then processing is simplified, but flexibility deteriorates when handling arbitrary-shaped moving objects resulting in inability to successfully in-paint regions
Solution Approach 1:
The patent implements dynamic content encoders that can adapt their shape and structure to match the arbitrary shapes of moving objects. Instead of using fixed geometric masks, the invention employs learnable deformation fields and adaptive mask generation through neural networks. These dynamic encoders can morph their receptive fields and segmentation boundaries to conform to any object shape, whether rectangular, circular, or irregular, while maintaining efficient processing through parameterized transformations rather than rigid geometric constraints
Solution Approach 2:
The patent creates a universal content encoder architecture that can handle multiple object shapes and sizes through a single unified model. The invention uses shape-agnostic feature extraction mechanisms that work equally well for objects of any geometry, eliminating the need for shape-specific processing pipelines. The universal encoder learns to represent diverse object shapes through learned embeddings and adaptive pooling operations, providing both implementation simplicity and shape flexibility simultaneously through a single multi-functional system
Data Source
AI summary
The present disclosure describes systems, non-transitory computer-readable media, and methods for accurately and efficiently removing objects from digital images taken from a camera viewfinder stream. For example, the disclosed systems access digital images from a camera viewfinder stream in connection with an undesired moving object depicted in the digital images. The disclosed systems generate a temporal window of the digital images concatenated with binary masks indicating the undesired moving object in each digital image. The disclosed systems further utilizes a generator as part of a 3D to 2D generative adversarial neural network in connection with the temporal window to generate a target digital image with the region associated with the undesired moving object in-painted. In at least one embodiment, the disclosed systems provide the target digital image to a camera viewfinder display to show a user how a future digital photograph will look without the undesired moving object.