Video Quality Assessment With Reduced Spatiotemporal Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video quality assessment (VQA) models face high computational complexity due to increasing video stream resolutions and frame rates, leading to impractical processing demands and inefficiencies in predicting perceptual quality, especially when traditional sampling techniques fail to capture quality-determinative regions and sequences effectively.
Innovation Solution
A two-stage, multiple dataset sampling technique is employed to evaluate spatiotemporal quality, using reduced-quality streams that maintain either the original temporal or spatial resolution, with patches sampled from these streams processed through a machine learning model comprising convolutional and self-attention neural networks to identify quality-relevant patches and improve video quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional VQA models process high-resolution video streams, then measurement precision of quality assessment is improved, but computational complexity increases exponentially
Solution Approach 1:
The video stream is divided into multiple spatio-temporal patches (STPs) that are sampled from different regions and time points. This segmentation allows the model to process smaller, manageable units while collectively covering the entire video content, reducing computational complexity while maintaining assessment accuracy.
Solution Approach 2:
Different regions of the video stream are sampled at different qualities and resolutions. The model processes these locally varied patches with appropriate computational resources, focusing detailed analysis on quality-critical regions while using coarser sampling in less critical areas, thereby optimizing the balance between precision and complexity.
2Measurement precision
If video stream resolution and frame rate are increased, then perceptual quality assessment accuracy is improved, but processing time and deployment cost increase
Solution Approach 1:
The model employs periodic sampling of spatio-temporal patches from the video stream rather than continuous processing. By strategically selecting patches at specific intervals and locations, the system achieves accurate quality assessment while significantly reducing processing time and computational resource requirements.
3Reliability
If full video stream is processed for quality assessment, then reliability of quality score is improved, but productivity of processing decreases
Solution Approach 1:
The model extracts and processes only the most relevant spatio-temporal patches from the video stream that contain quality-determinative information. By taking out and focusing on these critical regions rather than processing the entire video stream, the system maintains reliable quality assessment while dramatically improving processing efficiency and throughput.
Data Source
AI summary
A method for generating, for a video stream of a first spatial resolution and a first temporal resolution, a first reduced quality steam of a second spatial resolution and a second reduced-quality stream of a second temporal resolution. A first subset of STPs is sampled from the first reduced-quality stream and a second subset of STPs is sampled from the second reduced-quality stream. Using a machine learning model (MLM) the STPs are processed to identify a quality score for each quality-representative STPs that are representative of a quality of the video stream. One or more quality-improving actions for the video stream are identified using the quality scores of the quality-representative STPs.