Multi-Source Query Routing Using Generative AI and Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems require users to know the content, schema, and query methods of each data source, and they need extensive training with terabytes of data, which is not scalable for numerous disparate data sources, especially when privacy or access restrictions are involved, leading to inefficiencies and privacy concerns.
Innovation Solution
A generative artificial intelligence (GAI) system is used to parse natural language queries into simple questions, identify appropriate data sources, and generate custom queries in the correct format, without requiring extensive training on each data source or sharing sensitive data, enabling seamless access across heterogeneous data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional systems query multiple heterogeneous data sources individually, then data can be retrieved from each source, but users must know the format and language requirements for each data source, increasing system complexity and difficulty of operation
Solution Approach 1:
The patent introduces an intermediary layer (the querying system with GAI) between the user and multiple heterogeneous data sources. This intermediary automatically handles query translation, format conversion, and data source selection, allowing users to access data without knowing the specific requirements of each data source while the system manages the complexity of interfacing with multiple sources
2Adaptability or versatility
If AI models are trained with large amounts of data from diverse sources, then the model can answer queries across multiple sources, but privacy and security concerns arise when dealing with sensitive information
Solution Approach 1:
The patent extracts only the necessary metadata and schema information from data sources for training the GAI model, rather than using the actual sensitive data content. This allows the model to learn query patterns and data source characteristics without exposing or storing sensitive information, thereby maintaining model versatility while mitigating privacy risks
Solution Approach 2:
The system performs preliminary actions by pre-processing and anonymizing data source metadata before training the GAI model. This preliminary preparation allows the model to be trained on diverse data source characteristics without ever encountering or storing actual sensitive data, enabling versatile query handling while protecting privacy from the outset
Data Source
AI summary
Embodiments of the disclosed technologies include, in response to receiving a query, matching the query to metadata from a plurality of heterogeneous data sources, and selecting one or more data sources from the plurality of heterogeneous data sources for answering the query, by sending the query and embeddings of the matched metadata to a generative artificial intelligence (GAI), and prompting the GAI to select matching data sources. Based on the data from the GAI, generating one or more custom queries targeted to the matching data sources selected by the GAI, the custom queries formatted to be sent to the selected data sources, executing the one or more custom queries across the selected data sources, and summarizing results from the executing and providing a response to the query.