Video Matching Messaging App Visual Codebook Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face inefficiencies in finding and selecting appropriate images for messaging applications, as they must manually search through numerous options, which is time-consuming and discourages the use of image-based communication, leading to resource wastage.
Innovation Solution
A messaging application system that automatically selects and presents augmented reality experiences by analyzing a video captured by the user, identifying features and matching them with visual codebook clusters to find candidate videos and select relevant AR experiences, reducing the need for manual object specification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If users manually search through numerous images to find appropriate graphics for messaging applications, then they can select from a wide variety of options, but the process becomes time-consuming and discourages use of image-based communication
Solution Approach 1:
The system performs automatic image selection and AR experience matching without requiring manual user input. The messaging application captures video frames, extracts features, matches them with codebook clusters, and automatically presents relevant AR experiences, allowing the system to serve itself rather than requiring users to manually search through images
Solution Approach 2:
The system pre-processes video frames by extracting features and matching them with visual codebook clusters before the user needs to select an image. This preliminary automated analysis prepares candidate AR experiences in advance, eliminating the need for users to manually browse through numerous image options
2Productivity
If the system automatically analyzes video frames and matches features with visual codebook clusters to identify candidate videos, then the time and resources required are reduced, but the system complexity increases
Solution Approach 1:
The system introduces visual codebook clusters as an intermediary layer between raw video frame features and final AR experience selection. Features extracted from video frames are matched against pre-defined codebook clusters, which serve as a mediator to simplify the matching process and reduce computational complexity while maintaining high productivity
Solution Approach 2:
The system segments the image selection process into distinct modular components: video frame capture, feature extraction, codebook cluster matching, candidate video identification, and AR experience selection. This segmentation allows each component to be optimized independently, reducing overall system complexity while improving productivity
Data Source
AI summary
Aspects of the present disclosure involve a system and a method for performing operations comprising: identifying a plurality of features for frames of a video received by a messaging application server; assigning a first sequence of a first subset of the plurality of features and a second sequence of a second subset of the plurality of features respectively to a first nearest visual codebook cluster and a second nearest visual codebook cluster; applying the first and second nearest visual codebook clusters to a visual search database to identify a plurality of candidate matching videos; selecting a given matching video from the plurality of candidate matching videos based on a rank representing similarity of the given matching video to the video received by the messaging application server; and accessing an augmented reality experience corresponding to the given matching video.