Adaptive Multimodal Information Retrieval With Modality-Specific Chunking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems struggle to effectively handle multimodal data due to the lack of adaptive strategies and specialized models, leading to suboptimal performance and limited applicability across diverse data types.
Innovation Solution
A comprehensive framework for multimodal information retrieval is provided, involving vector creation, adaptive chunking, retrieval, and result surfacing, which optimizes chunking techniques for specific data modalities and includes metadata annotation to enhance the efficiency and accuracy of Large Language Models (LLMs) using Retrieval-Augmented Generation (RAG).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If uniform chunking strategies are used for all data types, then the system is simple to implement, but the retrieval performance deteriorates across diverse data types
Solution Approach 1:
The system dynamically adjusts chunking strategies based on data modality. Different chunking approaches (fixed-size, recursive, semantic, hierarchical) are selected and applied according to the specific characteristics of each data type (text, code, images, audio, video), allowing the system to adapt rather than use a static uniform approach
Solution Approach 2:
The patent applies different chunking quality standards to different data modalities. Each modality receives a customized chunking strategy optimized for its specific properties, such as using semantic chunking for text based on sentence boundaries and hierarchical structures, while using different approaches for images and audio data
2Device complexity
If generic models are used for information retrieval, then the system complexity is reduced, but the applicability across diverse data types is limited
Solution Approach 1:
The system segments the information retrieval task into distinct processing pipelines for different data modalities. Each modality (text, code, images, audio, video) has its own specialized processing path with appropriate chunking and retrieval strategies, allowing the system to handle diverse data types effectively
Solution Approach 2:
The patent creates a universal retrieval framework that can handle multiple data modalities through a common architecture. The system uses a unified search interface and result aggregation mechanism that works across all modalities, while incorporating modality-specific processing components
3Loss of information
If LLMs ingest all document tokens, then complete information is available, but the token waste on less relevant portions reduces retrieval efficiency
Solution Approach 1:
The system extracts only the relevant portions of documents for LLM processing. Through modality-specific chunking strategies, the system identifies and extracts meaningful units (sentences, paragraphs, semantic segments) that are most relevant to the query, excluding irrelevant portions from the token intake
Solution Approach 2:
The system performs preliminary processing of documents before they reach the LLM. Documents are pre-chunked into meaningful segments with metadata annotation, allowing the retrieval system to efficiently select and present only the most relevant chunks to the LLM, rather than feeding all document content
Data Source
AI summary
At least one processor can generate a classification of a document including a plurality of sections according to at least one of a structure of at least one of the plurality of sections, a hierarchy of the plurality of sections, and a content of at least one of the plurality of sections. The at least one processor can determine a chunking strategy optimized for the classification from among a plurality of available chunking strategies, divide the document into a plurality of chunks according to the chunking strategy, and generate respective vector representations of respective ones of the plurality of chunks. At least one database can contain the vector representations and supply the vector representations to retrieval augmented generation (RAG) operations performed by at least one large language model (LLM).