Image-Enhanced Document Search Embeddings for LLM Visual Context
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models like LLMs are limited by prompt size constraints, unable to include images in their input and output, and provide text-only answers, which are inadequate for documents larger than the prompt size limit, such as user guides and knowledge bases.
Innovation Solution
Index images relative to text chunks, generate image-enhanced embeddings by inserting placeholders into word embeddings, and process queries using generative AI models to incorporate relevant images into responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If documents are broken into chunks to fit prompt size limitations, then the documents can be processed by LLMs, but the images associated with the text chunks are lost and cannot be retrieved
Solution Approach 1:
The patent introduces image placeholders as intermediary elements that bridge the text chunks and the original images. These placeholders are embedded within the text chunk embeddings and serve as references that allow the system to retrieve and associate the correct images with their corresponding text portions during query processing, thus preventing image loss while maintaining prompt size constraints.
Solution Approach 2:
The patent transitions from a single-dimension text-only representation to a multi-dimensional representation by integrating image placeholders into the text embeddings. This creates a hybrid structure that combines text data with image references, enabling the system to handle both textual and visual information within the constrained prompt space.
2Ease of operation
If LLMs are used to provide text-only answers, then the response generation is simple, but the answers lack visual context and comprehensiveness
Solution Approach 1:
The patent merges text processing and image retrieval operations into a unified system. By integrating image placeholders directly into the text chunk embeddings and processing them together through the LLM, the system combines the simplicity of text-only generation with the benefit of visual context, allowing images to be naturally incorporated into the response workflow without adding operational complexity.
3Loss of information
If images are included in document processing, then the responses become more comprehensive, but the system complexity increases due to additional indexing and retrieval mechanisms
Solution Approach 1:
The patent creates simplified copies of image references in the form of placeholders that are embedded within text embeddings. Instead of implementing complex image processing and storage systems, the patent uses lightweight placeholder tokens that copy the essential reference information, dramatically reducing system complexity while maintaining the ability to retrieve and display relevant images.
Data Source
AI summary
A method, computer program product, and computing system for generating a plurality of chunks for a plurality of text portions of a document, wherein the document includes the plurality of text portions and a plurality of images. Each chunk is indexed using a word embedding. Each of the plurality of images is indexed based upon, at least in part, a position of a respective image relative to a corresponding chunk. An image placeholder is generated for each of the plurality of images. A plurality of image-enhanced embeddings is generated by inserting the image placeholder for each of the plurality of images into a respective word embedding for the corresponding chunk. The plurality of image-enhanced embeddings are provided for processing a query using a generative artificial intelligence (AI) model.