Compositing Aware Image Search Using Two-Stream CNN
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional digital image compositing techniques are computationally inefficient and user-frustrating due to reliance on manual feature extraction and single characteristic matching criteria, which fail to adapt to diverse characteristics and importance variations across different image categories, especially when dealing with large datasets like stock image systems.
Innovation Solution
A compositing aware digital image search system employs a two-stream convolutional neural network (CNN) to jointly learn feature embeddings from foreground and background images, using triplets with segmentation mask annotations and a triplet loss function to define compatibility, enabling adaptation to various characteristics and scalability with millions of images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional techniques use manual feature extraction and single characteristic matching, then the system is simpler to implement, but it is computationally inefficient and cannot adapt to diverse characteristics across different image categories
Solution Approach 1:
The patent replaces manual feature extraction and single-characteristic matching (mechanical/conventional methods) with a deep learning-based two-stream CNN system. The first stream extracts foreground features while the second stream extracts background features, enabling automatic adaptation to diverse characteristics across different image categories without manual intervention for each characteristic type.
Solution Approach 2:
The two-stream CNN system serves multiple functions simultaneously: it handles diverse image categories (foreground objects, backgrounds, composites), extracts multiple types of features automatically, and adapts to different characteristics (color, texture, shape, semantics) within a single unified framework, making the system universally applicable across various compositing tasks.
2Quantity of substance
If conventional techniques process large datasets like stock image systems, then more images are available for compositing, but the computational efficiency deteriorates and user interaction becomes frustrating
Solution Approach 1:
The system performs preliminary feature extraction and embedding generation for all images in the large dataset during an offline preprocessing stage. The two-stream CNNs extract foreground and background features separately, and compatibility scores are pre-computed and stored. This allows the system to handle millions of images without computational inefficiency during online user interactions, as the heavy processing is done in advance.
3Adaptability or versatility
If conventional techniques focus on a single characteristic or rely on manual extraction, then the system is easier to implement, but it fails to define matching criteria that adapt to different object categories and their varying importance
Solution Approach 1:
The patent replaces manual extraction and single-characteristic focus with automated two-stream CNN processing. The first stream processes foreground images to extract object features, while the second stream processes background images to extract scene features. The system automatically determines matching criteria by comparing features from both streams, adapting to different object categories and their varying importance without manual configuration.
4Measurement precision
If the system uses a two-stream CNN to jointly learn feature embeddings, then compatibility scoring becomes more accurate and adaptive, but the computational resources required increase
Solution Approach 1:
The patent segments the feature extraction process into two independent streams: one dedicated to foreground objects and another to backgrounds. This segmentation allows each stream to specialize in extracting relevant features for its specific task, improving compatibility scoring accuracy while enabling parallel processing that can optimize computational resource usage compared to a single monolithic model.
Data Source
AI summary
Compositing aware digital image search techniques and systems are described that leverage machine learning. In one example, a compositing aware image search system employs a two-stream convolutional neural network (CNN) to jointly learn feature embeddings from foreground digital images that capture a foreground object and background digital images that capture a background scene. In order to train models of the convolutional neural networks, triplets of training digital images are used. Each triplet may include a positive foreground digital image and a positive background digital image taken from the same digital image. The triplet also contains a negative foreground or background digital image that is dissimilar to the positive foreground or background digital image that is also included as part of the triplet.