In-Memory Vector Database for Scalable RAG Ingestion and Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems face challenges in efficiently storing and searching large numbers of dense vector embeddings to support generative artificial intelligence and retrieval-augmented generation processes, particularly in handling large volumes of data from diverse document sources.
Innovation Solution
Utilizing an in-memory data grid as a vector database with linearly-scalable data ingestion, enabling parallel processing of content ingestion and vector similarity searches, and supporting various document sources through HTTP URLs and cloud storage services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional database systems are used to store and search vector embeddings, then data persistence is ensured, but search speed and scalability deteriorate
Solution Approach 1:
The system segments data storage between memory (for active vector embeddings requiring fast access) and persistent storage (for long-term data retention). The in-memory data grid holds frequently accessed vector embeddings while maintaining the ability to persist data to external storage systems, thus achieving both fast search speeds and data persistence.
Solution Approach 2:
The in-memory data grid acts as an intermediary layer between the application and persistent storage systems. It provides fast in-memory access to vector embeddings while handling data loading and caching from persistent storage, thus mediating between the speed requirements of search operations and the persistence requirements of data storage.
2Productivity
If more computing hardware resources are allocated for parallel processing, then processing throughput increases, but system complexity and resource management difficulty increase
Solution Approach 1:
The in-memory data grid provides multiple functions within a single system: data storage, parallel processing coordination, result aggregation, and integration with retrieval-augmented generation frameworks. This multi-functionality allows the system to handle complex parallel processing tasks without proportionally increasing system complexity.
3Adaptability or versatility
If data is stored in distributed memory across multiple nodes, then scalability improves, but data consistency and access latency worsen
Solution Approach 1:
The system merges distributed in-memory storage with persistent storage backends, combining the scalability and speed of in-memory computing with the consistency and reliability of persistent storage. Data is replicated across multiple nodes in the in-memory grid while maintaining synchronization with the persistent storage layer.
Data Source
AI summary
In accordance with an embodiment, described herein are systems and methods for use of an in-memory data grid as a vector database, with linearly-scalable data ingestion, for use in generative artificial intelligence (AI), data visualization, or other applications that include the use of a large language model (LLM) or a retrieval-augmented generation (RAG) process. In accordance with an embodiment, where AI-related tasks or processes, such as content ingestion and vectorization, or vector similarity searches, can be performed in parallel, the in-memory data grid provides efficient scaling and execution of such processes. When tasked with large amounts of content to be vectorized—for example in a cloud environment or as part of an on-premise solution—the system can scale its processing of the content, in parallel where indicated, to perform an optimal utilization of available computing hardware resources, and expeditiously perform required tasks or processes.