Low-FPS Video Reconstruction Using Optical Flow Alignment Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Capturing and streaming high-quality videos at high frame rates consumes significant power and bandwidth, which is challenging for compact devices like smart watches and AR glasses, and traditional methods for framerate increase assume continuous motion, which is not applicable in cases of significant frame dropping.
Innovation Solution
Capture videos at a lower framerate on the camera and perform video frame interpolation on a receiver computer using optical flow, alignment, and fusion algorithms to reconstruct the video at a higher framerate, redistributing the computational burden from the camera to the receiver.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the camera captures and streams video at high framerate, then video quality and motion smoothness are improved, but power consumption and bandwidth requirements increase significantly
Solution Approach 1:
The system segments the video processing workload between the camera device and the receiver computer. The camera captures only low-framerate video data, while the receiver computer performs the computationally intensive frame interpolation to generate high-framerate output. This segmentation allows the camera to operate at low power while still delivering high-quality high-framerate video through the receiver's processing capabilities.
2Speed
If the camera captures and streams video at high framerate, then video quality and motion smoothness are improved, but bandwidth requirements increase significantly
Solution Approach 1:
The system segments the video data transmission from the heavy computational processing. Only low-framerate video data is transmitted over the network, consuming minimal bandwidth. The high-framerate reconstruction is performed locally at the receiver computer, eliminating the need to transmit large volumes of high-framerate video data while still achieving the desired output quality.
3Productivity
If traditional motion interpolation methods are used on captured video, then intermediate frames are generated to increase framerate, but these methods assume continuous motion and fail when significant frame dropping occurs
Solution Approach 1:
The system performs preliminary actions by capturing video at the original low framerate with proper timing information preserved. The receiver computer then uses this timing information to accurately determine the number of intermediate frames needed for each interval between captured frames. This preliminary preservation of temporal information enables reliable reconstruction even when significant frame dropping has occurred, as the system can calculate the exact motion progression required.
Data Source
AI summary
In one embodiment, a method includes accessing a video captured by cameras which is associated with a first framerate lower than a threshold framerate, for any two adjacent frames of the accessed video: generating a warped frame from the two adjacent frames based on an optical flow associated with the two adjacent frames, determining alignments for the two adjacent frames, respectively, fusing the determined alignments for the two adjacent frames, and generating a reconstructed frame based on the fused alignment, and reconstructing the accessed video based on the any two adjacent frames and their respective reconstructed frames, wherein the reconstructed video is associated with a second framerate higher than the threshold framerate.