Personalized Video Cropping With Face-Aware Keyframing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The mismatch between the display aspect ratio and orientation of devices and the aspect ratio of media often results in letterboxing, reducing the video size and requiring manual or non-optimal cropping.
Innovation Solution
A method using a trained machine-learning model to determine personalized crop regions based on face detection, motion analysis, and aesthetic scores, adjusting scores to prioritize important faces, and generating crop keyframing to adapt video aspect ratio and orientation for optimal display on different devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual or non-optimal cropping is used to adapt video to device aspect ratio, then the video can be displayed without letterboxing, but the video quality and important content (faces) may be lost or reduced
Solution Approach 1:
The system automatically performs video cropping without user intervention by using machine learning models to detect faces and determine optimal crop regions. The algorithm independently analyzes video frames, identifies important content, and generates crop keyframing parameters, enabling the system to serve itself in adapting video to different device aspect ratios while preserving quality
Solution Approach 2:
The system dynamically adjusts crop region parameters (position, size, aspect ratio) based on detected face locations and device characteristics. By changing these parameters frame-by-frame or keyframe-by-keyframe, the system adapts the video display to match device aspect ratios while maintaining optimal framing of important content
2Manufacturing precision
If personalized video cropping with face detection is implemented, then important faces are preserved in the cropped video, but the computational complexity and processing time increase
Solution Approach 1:
The system performs face detection and crop region determination in advance during video processing or playback preparation. By pre-calculating crop keyframing parameters and storing them with the video, the complex face detection and analysis work is completed beforehand, reducing real-time processing requirements while ensuring accurate face preservation
Solution Approach 2:
The video processing is divided into discrete keyframes with associated crop parameters. Instead of processing every frame continuously, the system identifies key moments and determines crop regions for those keyframes, then interpolates between them. This segmentation reduces overall computational complexity while maintaining face preservation accuracy at critical moments
3Area of stationary object
If video is cropped to match device aspect ratio, then the video utilizes the full display area without black borders, but the original video composition and important content may be distorted or excluded
Solution Approach 1:
The system applies different crop regions to different time segments of the video based on local content analysis. Instead of using a single fixed crop region for the entire video, the algorithm determines optimal crop parameters for each keyframe based on the specific content present at that moment, ensuring that important faces and objects are preserved in their respective contexts while maximizing display area utilization
Data Source
Figure 1
Figure 2A~2B
Figure 3A~3B
AI summary
Described are methods, systems, and computer-readable media to automatically crop videos using personalized parameters. Some implementations include a computer- implemented method that comprises obtaining an input video, determining a per-frame crop score for one or more candidate crop regions in each frame of the input video, generating a face signal for the one or more candidate crop regions, adjusting each per-frame crop score based on the face signal, determining a minimal cost path that represents crop region locations based on motion cost and the adjusted per-frame crop score, generating crop keyframing corresponding to the crop region locations along the minimal cost path, wherein the crop keyframing includes a start frame, an end frame, and crop region location, and outputting a modified video that has one or more of an output aspect ratio or an output orientation that is different than a corresponding aspect ratio or an orientation of the input video.