Video Querying With Analytics Metadata for Faster LLM Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) are limited in their ability to efficiently and quickly analyze video clips due to memory and computational constraints, while video analytics methods can process video quickly but lack the ability to generate rich information about objects in the scene.
Innovation Solution
A system that combines video analytics and LLMs by using video analytics to generate compact metadata, such as object trajectories, which are then used to create prompts for the LLM to analyze, enabling efficient querying of video content with natural language.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If LLMs are used to analyze video clips directly, then rich information about objects can be generated, but processing speed decreases and computational resources are exhausted
Solution Approach 1:
The video analysis task is segmented into two parts: video analytics algorithms handle frame-level object detection and tracking, while LLMs handle high-level semantic reasoning. This division allows each component to operate within its optimal performance characteristics, avoiding the need for LLMs to process every frame directly.
Solution Approach 2:
Video analytics algorithms perform preliminary analysis of video frames to extract object information, trajectories, and visual features before presenting this processed data to the LLM. This preliminary action reduces the computational burden on the LLM while preserving essential information.
2Productivity
If video analytics algorithms are used to process video, then processing speed increases, but the ability to generate rich information about objects decreases
Solution Approach 1:
The system merges the outputs of video analytics algorithms (object detection, tracking, visual features) with the semantic reasoning capabilities of LLMs. This combination preserves the processing speed advantage of video analytics while adding the information richness of LLMs through the integrated prompt generation process.
3Loss of information
If LLMs process complete video sequences, then comprehensive understanding is achieved, but memory requirements exceed available resources
Solution Approach 1:
The system extracts only the essential information from video frames through video analytics algorithms, presenting a condensed representation to the LLM rather than the complete video sequence. This extraction reduces memory consumption while maintaining comprehensive understanding through the structured prompt format that includes object information, trajectories, and visual features.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and system for querying a video by incorporating video analytics and large language models (LLM). The method may include the following steps: receiving a video stream having a sequence of frames and including one or more objects; applying video analytics algorithms to the video stream, to yield video analytics features indicative of the one or more objects; receiving a user query comprising a verbal or auditory enquiry relating to the one or more objects in the video stream; carrying out a selection of at least one of the one or more objects in the video stream; generating, a prompt which is usable as a query for a large language model (LLM), based on: the user query, the video analytics features, and the selection of the at least one of the one or more objects; and applying, the prompt to the LLM, to yield an LLM response.