Federated Query Platform for Heterogeneous Data Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to multimodal data integration are suboptimal due to the need for manual query generation across disparate data types and locations, requiring knowledge of storage mechanisms and query languages, leading to inefficiencies in data storage and query performance.
Innovation Solution
A three-tier system with a knowledge-driven query and analysis platform that provides a single logical interface for interacting with diverse data types, abstracting away the need for knowledge of data locations and storage mechanisms, using a semantic domain model to link and query data across multiple data stores without moving data to a single location.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all different types of data are integrated into a single repository using ETL operations, then data integration is achieved, but data storage efficiency and query performance deteriorate due to forcing non-relational data into relational structures
Solution Approach 1:
The patent divides the data integration architecture into multiple specialized data stores (relational, NoSQL, file systems, data lakes) rather than forcing all data into a single repository. Each data store maintains its native format and optimization, allowing efficient storage and querying while achieving integration through a federated query layer that translates and coordinates queries across these segmented stores.
2Adaptability or versatility
If different types of data are collected into a single NoSQL data store, then data format flexibility is improved, but data storage efficiency and query performance worsen due to ignoring structural attributes
Solution Approach 1:
The patent introduces a federated query layer as an intermediary between the user and multiple specialized data stores. This mediator translates high-level queries into store-specific queries, manages data retrieval across different formats, and returns unified results. This allows the system to maintain data format flexibility while achieving high query performance by leveraging the structural optimizations of each specialized store through intelligent query translation and coordination.
3Ease of operation
If manual query generation is used to access disparate data types across distributed sources, then data access flexibility is maintained, but user burden and infrastructure complexity increase
Solution Approach 1:
The patent creates a universal federated query layer that provides a single, unified interface for accessing diverse data types across distributed sources. This multi-functional query system handles translation to multiple data store formats, coordinates queries across distributed locations, and manages result aggregation automatically. Users interact with a single standardized interface rather than learning multiple query languages, while the infrastructure complexity is encapsulated within the federated query layer rather than being exposed to users.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A system to query a federated store containing disparate data types and stores, the system including a UI or API to specify query details, a metadata knowledge graph with metadata describing the contents of the data stores, the relationships among them, and how to programmatically query the data stores, a predefined constrainable query ('nodegroup') store containing nodegroups providing a template to search the data stores, the querying layer including services and libraries to process nodegroups and generate a set of queries, a query and analysis platform providing the set of queries to at least one data store for execution at the federated stores and return a result, a scalable analytic execution layer applying machine learning/artificial intelligence techniques to analyze the query results and presenting data analysis result visualizations. A method and a non-transitory medium are also disclosed.