Controllable Still Image Animation via Refined Dense Optical Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies struggle to efficiently convert a single still image into a high-quality video with user control over the animation, as they often produce short, low-resolution videos with cross-fading artifacts.
Innovation Solution
A machine learning-based system that includes a fluid animation function, which generates an animated video from a single still image by animating fluid elements using a refined dense optical flow and flow field pairs, allowing for user control over animation direction and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If existing photo animation software is used to generate animated video from still images, then animation effect is provided, but the video duration is limited and resolution is low
Solution Approach 1:
The patent segments the video generation process into multiple independent components: optical flow estimation, flow field generation, and frame synthesis. Each component processes independently, allowing the system to generate unlimited frames without compounding errors, thus extending video duration while maintaining high resolution.
Solution Approach 2:
The patent performs preliminary action by pre-computing the optical flow and flow fields from the input still image before generating video frames. This pre-computation stores the motion information in advance, enabling the system to generate frames of any duration by simply replaying the pre-computed flow fields, thus extending video duration without sacrificing resolution.
2Ease of manufacture
If existing photo animation software is used, then animation is generated, but cross-fading artifacts appear in the output
Solution Approach 1:
The patent replaces the traditional mechanical interpolation method (cross-fading) with a physics-based optical flow approach. Instead of blending frames, the system computes realistic motion vectors and flow fields that naturally describe fluid element movement, eliminating cross-fading artifacts while maintaining ease of animation generation.
Solution Approach 2:
The patent introduces an intermediary optical flow computation step between the input still image and the output video frames. This intermediary layer computes the motion patterns of fluid elements, serving as a mediator that translates static image data into realistic animated sequences without direct frame interpolation, thus eliminating artifacts.
3Duration of action of moving object
If more video frames are generated from a single still image, then video duration increases, but computing resources increase
Solution Approach 1:
The patent performs all computationally intensive operations (optical flow estimation, flow field generation) in advance from the input still image. The pre-computed flow fields are stored and can be replayed to generate frames of any duration without additional heavy computation, thus extending video duration while minimizing computing resource usage during frame generation.
Solution Approach 2:
The patent uses the pre-computed flow fields as templates that can be copied and applied to generate multiple video frames. Instead of re-computing motion information for each frame, the system copies and applies the same flow field data, significantly reducing computing resources while enabling unlimited video duration.
Data Source
AI summary
Systems and methods for machine learning based controllable animation of still images is provided. In one embodiment, a still image including a fluid element is obtained. Using a flow refinement machine learning model, a refined dense optical flow is generated for the still image based on a selection mask that includes the fluid element and a dense optical flow generated from a motion hint that indicates a direction of animation. The refined dense optical flow indicates a pattern of apparent motion for the at least one fluid element. Thereafter, a plurality of video frames is generated by projecting a plurality of pixels of the still image using the refined dense optical flow.