Word Image Embedding via Shared Vector Space
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text recognition and image retrieval systems fail to effectively consider semantic information, leading to inaccuracies in word image recognition and retrieval, especially when words are not cropped exactly or when out-of-vocabulary words are encountered, and they do not provide a compact representation of word images that encodes semantics.
Innovation Solution
A method and system that embeds word images and semantic concepts into a common vectorial space using a convolutional neural network (CNN) trained end-to-end, allowing for direct computation of relevance scores between images and concepts without intermediate transcription, enabling semantic recognition and retrieval tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a two-step approach (transcription followed by lexical matching) is used to achieve semantic recognition, then semantic concepts can be assigned to word images, but the system fails when words are not cropped exactly or when out-of-vocabulary words are encountered
Solution Approach 1:
The patent divides the semantic recognition task into two independent embedding processes: one for visual features (image embedding) and one for semantic features (concept embedding). These segmented processes operate in separate spaces and are then aligned through contrastive learning, allowing the system to handle cases where traditional transcription fails.
Solution Approach 2:
The patent introduces a contrastive learning mechanism as an intermediary that bridges the visual feature space and semantic concept space. This intermediary aligns the two spaces by minimizing the distance between matching image-concept pairs while maximizing the distance between non-matching pairs, enabling robust semantic recognition without relying on accurate transcription.
2Measurement precision
If traditional text recognition methods are used, then exact word transcription can be achieved, but the system cannot capture semantic meanings or perform concept-based retrieval
Solution Approach 1:
The patent transitions from one-dimensional text-based semantic representation to a multi-dimensional embedding space where both visual and semantic features coexist. By projecting both word images and concepts into a shared vector space, the system preserves semantic information while enabling direct comparison and retrieval operations.
Solution Approach 2:
The patent changes the parameter representation from discrete text tokens to continuous vector embeddings. This parameter transformation allows the system to capture semantic relationships through geometric relationships in the embedding space, enabling concept-based retrieval and semantic analysis beyond exact string matching.
3Measurement precision
If exact word matching is used for retrieval, then precise transcription retrieval is achieved, but semantic-based retrieval and classification cannot be performed
Solution Approach 1:
The patent creates a universal embedding space that serves multiple functions: exact word matching, semantic similarity search, concept-based retrieval, and classification. By aligning visual and semantic features in a shared space, the system can perform various retrieval tasks using the same infrastructure, achieving both precision and versatility.
Data Source
AI summary
A system and method enable semantic comparisons to be made between word images and concepts. Training word images and their concept labels are used to learn parameters of a neural network for embedding word images and concepts in a semantic subspace in which comparisons can be made between word images and concepts without the need for transcribing the text content of the word image. The training of the neural network aims to minimize a ranking loss over the training set where non relevant concepts for an image which are ranked more highly than relevant ones penalize the ranking loss.