Query Communication Mesh for Distributed Data Access Without Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle to efficiently manage and query large volumes of data distributed across multiple data sources, including those behind firewalls and in the cloud, due to scalability issues and security concerns, while traditional methods like ETL replication create redundancy and legal barriers.
Innovation Solution
A query communication mesh system using enterprise query base nodes to distribute queries directly to data sources, optimizing sub-queries based on metadata and network security, and aggregating results without storing data, enabling access to diverse data sources through a directional graph or tree-like mesh.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed across multiple data sources for security and availability, then reliability and availability are improved, but query complexity and system complexity increase
Solution Approach 1:
The patent introduces a query processor as an intermediary component that receives queries from clients, translates them into data access requests, and coordinates retrieval operations across multiple distributed data sources. This mediator abstracts the complexity of accessing scattered data from the end user, allowing reliable data availability while simplifying the query interface.
Solution Approach 2:
The system segments data access operations by distributing queries to different nodes or data sources based on data location and access requirements. The query processor divides complex queries into smaller sub-queries that can be executed independently across multiple data sources, improving reliability while managing complexity through modular processing.
2Ease of operation
If traditional ETL replication methods are used to access data, then data accessibility is improved, but data redundancy and storage requirements increase
Solution Approach 1:
Instead of replicating entire datasets through ETL processes, the patent extracts only the specific data needed for each query from its original location in distributed data sources. This selective extraction approach maintains data accessibility for users while eliminating unnecessary data redundancy and reducing storage requirements.
Solution Approach 2:
The system employs dynamic data access patterns where queries are routed to appropriate data sources based on real-time needs rather than static data replication. This dynamic approach allows the system to access data on-demand from its original locations, improving accessibility without creating redundant copies.
3Productivity
If queries are distributed to multiple nodes for performance, then query processing speed is improved, but network communication overhead increases
Solution Approach 1:
The query processor distributes queries to multiple nodes only when necessary based on data location and query requirements. Rather than always distributing to maximize speed, the system applies partial distribution strategies that balance performance gains against network communication overhead, sending queries to the minimum number of nodes needed for efficient data retrieval.
Data Source
AI summary
A computing system for querying multiple data sources and method therefor is provided. The computing system may comprise one or more nodes in communication with at least one data source of the multiple data sources to access data therefrom. The computing system may further comprise a second node in communication with the one or more nodes. The second node may be configured to receive a query instance and process the query instance to generate one or more relational query instances. The one or more relational query instances may be distributed among the nodes to extract data from the at least one data source in communication therewith corresponding to the respective one or more relational query instances. The second node may be further configured to receive extracted data from each of the one or more nodes queried. The second node may be further configured to aggregate the extracted data.