Federated Query Platform Using Knowledge Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to integrating diverse data types across distributed data stores are burdensome, requiring users to have knowledge of data locations, storage mechanisms, and query languages, and do not provide a mechanism to describe the relationships between different datasets, leading to suboptimal performance and increased network traffic.
Innovation Solution
A three-tier system with a knowledge-driven query and analysis platform that provides a single logical interface for interacting with diverse data, using a knowledge graph to model datasets and relationships, allowing users to run analytics without knowing the underlying data storage locations or mechanisms, and enabling data to be stored in datatype-appropriate repositories, thus avoiding data movement and optimizing query performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all different types of data are shoehorned into a common format within a single repository (ETL operations into a large data warehouse), then data integration is achieved, but data storage efficiency and query performance deteriorate
Solution Approach 1:
The system segments the monolithic data warehouse approach into multiple specialized data stores, each optimized for specific data types (relational data, time-series data, unstructured data, etc.). Instead of forcing all data into a single repository, the patent divides data storage across heterogeneous storage systems that maintain their native formats and optimizations.
Solution Approach 2:
Each data store in the federated system is configured with local quality optimizations specific to its data type. For example, time-series databases use time-based indexing, relational databases use schema-based optimizations, and unstructured data stores use document-based optimizations. This allows each storage system to operate at peak efficiency for its intended data type.
2Adaptability or versatility
If different types of data are collected into a single NoSQL data store, then data format flexibility is improved, but data storage efficiency and query performance deteriorate
Solution Approach 1:
The system segments the single NoSQL store approach into multiple specialized data stores, each with appropriate data structures and indexing for its specific data type. This segmentation allows each store to optimize for its workload while the federation layer provides unified access.
Solution Approach 2:
The patent introduces a federation layer as an intermediary between the user and the distributed data stores. This mediator translates unified query interfaces into store-specific queries, allowing users to access heterogeneous data sources through a consistent interface without sacrificing the performance optimizations of each individual store.
3Adaptability or versatility
If complex middleware is built to query across diverse datastores using a common query language, then data integration capability is improved, but system complexity and user knowledge requirements increase
Solution Approach 1:
The federation layer provides universal access to multiple diverse data stores through a common query interface. It implements multi-functionality by supporting various query languages and data models while presenting a unified interface to users, eliminating the need for store-specific query knowledge.
Solution Approach 2:
The patent uses a federation layer as an intermediary that handles the complexity of cross-datastore queries. This mediator translates high-level queries into store-specific operations, managing the complexity internally while presenting a simple interface to users. The federation layer automatically handles data location, format conversion, and query optimization.
4Adaptability or versatility
If users manually generate multiple queries to interrogate disparate data types across distributed sources, then data access flexibility is improved, but time consumption and infrastructure burden increase
Solution Approach 1:
The federation layer provides self-service capabilities by automatically discovering available data sources, their schemas, and relationships. Users can query distributed data through a unified interface without manually configuring connections or learning multiple query languages. The system automatically routes queries to appropriate data stores and integrates results.
Data Source
AI summary
A system to generate and run federated queries against a plurality of data stores storing disparate data types, the system including a user interface receiving query details from a data consumer, a metadata knowledge graph containing metadata for links and relationships of the data stores, a knowledge-driven querying layer accessing the graph and selecting predefined constrainable queries from a nodegroup store and applying the metadata links/relationships to the predefined constrainable queries to assemble subqueries, a query and analysis platform providing the subqueries to some of the data stores for execution, a scalable analytic execution layer receiving and aggregating search results from the data stores into a merged search result and/or obtaining analytic results by applying machine learning and artificial intelligence techniques to the distributed data, the user interface presenting visualizations generated from the merged search results, and/or the analytic results. A system and a non-transitory computer-readable medium are also disclosed.