Visio-linguistic Feature Composition for Text-Conditioned Image Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-conditioned image search systems are unable to capture detailed user requirements that span multiple levels of granularity, limiting their ability to understand subtleties in content and style features, and fail to efficiently compose visual representations with natural-language semantics.
Innovation Solution
The techniques employ a process of generating multiple visual transformations over varying levels of granularity to learn a visio-linguistic representation of the source image and text query, using neural networks to decompose images and text into feature vectors at different levels of granularity, and then compose a visio-linguistic representation for improved image search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single input image and keywords are used for text-conditioned image search, then the system is simple to operate, but it cannot capture detailed user requirements spanning multiple levels of granularity
Solution Approach 1:
The patent segments the image into multiple levels of granularity (fine-grained, intermediate-grained, and coarse-grained levels) and processes each level separately through dedicated neural network branches. This segmentation allows the system to capture detailed user requirements at different scales while maintaining manageable complexity through modular architecture.
Solution Approach 2:
The patent adds a granularity dimension to the feature extraction process by creating hierarchical feature representations at multiple scales. Instead of a single flat feature space, the system operates in a multi-dimensional granularity space, enabling precise capture of both local and global image characteristics.
2Reliability
If existing text-conditioned image search systems are used, then the system structure is simple, but it fails to understand subtleties in content and style features
Solution Approach 1:
The patent applies local quality by using different neural network processing paths for different granularity levels. Fine-grained features (local details) are processed with higher resolution networks, while coarse-grained features (global structure) use lower resolution networks, allowing each region to be processed with appropriate detail level.
Solution Approach 2:
The patent creates a composite feature representation by combining features from multiple granularity levels. The final image embedding is composed of fine-grained, intermediate-grained, and coarse-grained features, creating a multi-level composite representation that captures both detailed and holistic image characteristics.
3Measurement precision
If multiple levels of granularity are processed, then search accuracy improves, but computational complexity increases
Solution Approach 1:
The patent implements dynamic feature processing where the system adaptively processes features at different granularity levels based on the search query requirements. Not all granularity levels are processed with equal depth for every query, allowing computational resources to be dynamically allocated to the most relevant feature levels.
Solution Approach 2:
The patent reduces computational burden by projecting high-dimensional fine-grained features into lower-dimensional intermediate and coarse representations. This dimensional reduction across granularity levels maintains information integrity while significantly reducing the computational complexity of processing all features at full resolution.
Data Source
AI summary
Techniques are disclosed for text-conditioned image searching. A methodology implementing the techniques includes decomposing a source image into visual feature vectors associated with different levels of granularity. The method also includes decomposing a text query (defining a target image attribute) into feature vectors associated with different levels of granularity including a global text feature vector. The method further includes generating image-text embeddings based on the visual feature vectors and the text feature vectors to encode information from visual and textual features. The method further includes composing a visio-linguistic representation based on a hierarchical aggregation of the image-text embeddings to encode visual and textual information at multiple levels of granularity. The method further includes identifying a target image that includes the visio-linguistic representation and the global text feature vector, so that the target image relates to the target image attribute, and providing the target image as an image search result.