Natural Language Querying of Heterogeneous Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for querying data across multiple heterogeneous data sources require strong technical expertise and are difficult to use due to the complexity of interacting with different data systems and inconsistent naming schemes.
Innovation Solution
An analysis system that allows users to query heterogeneous data sources using natural language questions, utilizing a machine learning based language model to generate database queries and connect to various data sources, while maintaining privacy by not exposing metadata to external services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional querying techniques are used to interact with heterogeneous data sources, then data can be retrieved from multiple systems, but users require strong technical expertise and the system becomes difficult to use
Solution Approach 1:
The patent introduces a natural language processing interface as an intermediary between users and heterogeneous data sources. This mediator translates user-friendly natural language questions into database-specific queries, eliminating the need for users to learn multiple query languages and system-specific syntax while maintaining the ability to access diverse data sources
2Reliability
If expert knowledge is required to interact with complex data systems, then data retrieval can be achieved, but the system becomes inaccessible to non-experts
Solution Approach 1:
The patent creates a virtual copy or abstraction layer of the complex data source schemas. Instead of requiring users to understand the actual complex database structures, the system presents simplified metadata representations that mirror the data organization without exposing the underlying complexity, allowing non-experts to query accurately
3Adaptability or versatility
If consistent naming schemes are not used across data assets, then data can be stored flexibly in different systems, but query accuracy decreases due to confusion
Solution Approach 1:
The natural language processing system acts as an intermediary that translates ambiguous natural language queries into precise database queries. The system uses metadata about data assets to disambiguate references and map user intent to the correct data sources, maintaining query accuracy despite inconsistent naming schemes across heterogeneous systems
4Ease of operation
If machine learning language models are used to generate queries, then natural language querying is enabled, but generated data asset names may not correspond to actual data assets
Solution Approach 1:
The system implements a feedback mechanism where the generated database query is validated against actual data source schemas. When the language model generates hallucinated data asset names, the system detects these mismatches, retrieves relevant metadata, and uses it to correct the generated query, thereby maintaining reliability while preserving natural language accessibility
Data Source
AI summary
An analysis system connects to a set of data sources and perform natural language questions based on the data sources. The analysis system connects with the data sources and retrieves metadata describing data assets stored in each data source. The analysis system generates an execution plan for the natural language question. The analysis system finds data assets that match the received question based on the metadata. The analysis system ranks the data assets and presents the ranked data assets to users for allowing users to modify the execution plan. The analysis system may use execution plans of previously stored questions for executing new questions. The analysis system supports selective preprocessing of data to increase the data quality.