Combined Structure Style Network for Image Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image searching techniques limit users in finding visual artistic works by relying on text-based queries and predefined taxonomies, failing to retrieve images with desired visual characteristics, which hampers the creation of visual artistic content.
Innovation Solution
A combined structure and style network is employed, leveraging a triplet convolutional neural network to recognize both structure and style in images, allowing users to search for images with specific objects and styles through sketch-based queries, enabling the identification of images with desired visual features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional text-based search techniques are used, then search simplicity is maintained, but image retrieval precision deteriorates because users cannot specify desired visual characteristics
Solution Approach 1:
The search system is segmented into two independent components: a structure network for object recognition and a style network for visual characteristic recognition. This segmentation allows each network to specialize in one aspect, improving overall retrieval precision while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The combined structure and style network serves multiple functions: it can process text-based queries, visual example queries, and sketch queries simultaneously. This multi-functionality enables precise image retrieval across different query types without requiring separate systems for each query method.
2Measurement precision
If visual example search techniques are used, then search expressiveness is improved, but search accuracy deteriorates because the system returns near-duplicates or particular instances rather than images matching desired characteristics
Solution Approach 1:
The search system separates structure recognition and style recognition into independent networks. This allows the system to understand both the object content and visual characteristics independently, then combine them to return images that match desired characteristics rather than just near-duplicates of query examples.
Solution Approach 2:
The combined network acts as an intermediary that processes both structure and style information, mediating between the query input and image retrieval output. This intermediary layer enables the system to translate diverse query types into accurate image matches based on desired visual characteristics.
3Measurement precision
If sketch-based queries are used, then object specification precision is improved, but style specification capability is lost
Solution Approach 1:
The combined structure and style network provides universal functionality that handles both object specification through sketches and style specification through visual characteristics. The network can process sketch queries while simultaneously understanding style information from training images, enabling both precision object specification and style specification capability.
Data Source
AI summary
A combined structure and style network is described. Initially, a large set of training images, having a variety of different styles, is obtained. Each of these training images is associated with one of multiple different predetermined style categories indicating the image's style and one of multiple different predetermined semantic categories indicating objects depicted in the image. Groups of these images are formed, such that each group includes an anchor image having one of the styles, a positive-style example image having the same style as the anchor image, and a negative-style example image having a different style. Based on those groups, an image style network is generated to identify images having desired styling by recognizing visual characteristics of the different styles. The image style network is further combined, according to a unifying training technique, with an image structure network configured to recognize desired objects in images irrespective of image style.