Federated Query Mesh for Distributed Data Without Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle to efficiently manage and query large volumes of data distributed across multiple data sources, including those behind firewalls and in the cloud, due to scalability issues and security concerns, while traditional data replication methods are cumbersome and often prohibited by legal restrictions.
Innovation Solution
A query base system employing an elastic mesh technique that distributes queries directionally through enterprise query base nodes, translating and optimizing queries across different data sources, including those behind firewalls and in the cloud, using a hierarchical structure and natural language processing to access data without replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple data sources for security and availability, then data availability and security are improved, but query complexity and system management difficulty increase
Solution Approach 1:
The patent introduces an intermediary query processing layer that sits between the user and distributed data sources. This intermediary receives queries, translates them into source-specific query languages, and coordinates execution across multiple data sources, thereby reducing query complexity for users while maintaining data availability across distributed sources
Solution Approach 2:
The system segments the query processing function across multiple components: query parser, translator, executor, and result aggregator. Each component handles a specific aspect of query processing independently, making the overall complex system more manageable and maintainable while enabling efficient querying of distributed data
2Ease of operation
If traditional data replication methods are used to access data, then data accessibility is improved, but legal compliance and operational complexity worsen
Solution Approach 1:
The patent extracts the data access function from the data storage function. Instead of replicating data to multiple locations for accessibility, the system extracts queries to be executed at their original locations and aggregates results, eliminating the need for complex replication operations while maintaining accessibility
Solution Approach 2:
The system changes the parameter of data access from physical replication to logical federation. By transforming data access requests into coordinated queries across distributed sources rather than copying data, the system achieves accessibility without the operational complexity of replication management
3Device complexity
If queries are processed locally without distribution, then processing simplicity is maintained, but data freshness and system scalability worsen
Solution Approach 1:
The query processing system dynamically adapts its behavior based on data location and accessibility requirements. Rather than using a fixed local processing approach, the system dynamically determines the optimal execution path by considering data freshness requirements, network conditions, and source characteristics, thereby maintaining simplicity while ensuring data freshness
Data Source
AI summary
A computing system for querying multiple data sources and a method therefore is provided. The computing system may comprise one or more nodes in communication with at least one data source of the multiple data sources to access data therefrom. The computing system may further comprise a second node in communication with the one or more nodes. The second node may be configured to receive a query instance and process the query instance to generate one or more relational query instances. The one or more relational query instances may be distributed among the one or more nodes to extract data from the at least one data source in communication therewith corresponding to the respective one or more relational query instances. The second node may be further configured to receive extracted data from each of the one or more nodes queried. The second node may be further configured to aggregate the extracted data.