Text-Based Image Search Through Fused Visual-Textual Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing person search imagery methods require at least one-shot image samples, which is limiting when only a brief textual description of the target person is available, and conventional zero-shot learning techniques fail in unstructured surveillance data due to unreliable category name semantics and noisy conditions.
Innovation Solution
A machine learning model is developed that maps text attribute labels onto a larger vector space using Word2Vec and clusters similar images, enabling text-based searching without metadata by fusing visual and textual embeddings through hierarchical matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional zero-shot learning techniques are used, then text-based searching is possible without visual content analysis, but the search fails in unstructured surveillance data due to unreliable category name semantics and noisy conditions
Solution Approach 1:
The patent introduces an intermediary alignment mechanism that learns a shared embedding space for visual and textual attributes. This intermediary space bridges the gap between unreliable category names and visual content, enabling reliable text-based search in surveillance data by translating queries into the visual attribute space through learned correspondences rather than direct category matching
Solution Approach 2:
The patent transforms the search problem from category-based matching to attribute-based matching by changing the parameter space. Instead of relying on unreliable category semantics, the system extracts and matches visual attributes (e.g., clothing, physical characteristics) and textual attributes independently, then aligns them through learned transformations, making the search robust to noisy conditions and unstructured data
2Measurement precision
If existing person search methods are used, then visual matching is possible, but the methods require at least one-shot image samples which are not available when only textual description is provided
Solution Approach 1:
The patent creates a universal search system that handles multiple query types (image-based, text-based, and hybrid) through a unified attribute alignment framework. The system extracts attributes from both visual and textual inputs and performs matching in a shared space, making it versatile enough to work with text-only queries, image-only queries, or combined inputs without requiring separate specialized methods
Solution Approach 2:
The patent transitions from traditional image-space matching to an attribute-space matching by projecting both visual and textual queries into a shared embedding dimension. This dimensional transformation allows text descriptions to be compared with visual content through their attribute representations rather than requiring direct pixel-level comparison, enabling zero-shot text-based search
3Measurement precision
If attribute labelled training datasets are used, then text attribute matching is possible, but the breadth of textual attributes is limited by the labelled data available
Solution Approach 1:
The patent performs preliminary attribute extraction and alignment during the training phase, learning the correspondence between visual and textual attributes across multiple datasets. By pre-training the attribute alignment model on diverse labelled data, the system acquires robust attribute representations that can generalize to unlabelled surveillance data, expanding the effective range of searchable attributes beyond what any single dataset provides
Data Source
AI summary
Method and system for building a machine learning model for finding visual targets from text queries, the method comprising the steps of receiving a set of training data comprising text attribute labelled images, wherein each image has more than one text attribute label. Receiving a first vector space comprising a mapping of words, the mapping defining relationships between words. Generating a visual feature vector space by grouping images of the set of training data having similar attribute labels. Mapping each attribute label within the training data set on to the first vector space to form a second vector space. Fusing the visual feature vector space and the second vector space to form a third vector space. Generating a similarity matching model from the third vector space