Asynchronous Querying Language for Multi-Store Data Federation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tenant database systems face challenges in efficiently managing and querying large volumes of data from diverse data sources with different structures and architectures, leading to increased resource overhead and user dissatisfaction.
Innovation Solution
The implementation of an asynchronous querying language (QL) as an API that allows users to submit jobs for asynchronous execution, enabling federated querying across multiple data stores with support for persistent and ephemeral job types, and providing features like federated joins and aggregation, while also managing resource limits and error handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If synchronous querying is used to search large amounts of data across multiple data stores, then query results are obtained quickly, but resource overhead increases significantly
Solution Approach 1:
The system pre-generates materialized views from source data before queries are executed. These pre-computed results are stored and can be quickly retrieved when query requests are made, eliminating the need for resource-intensive real-time computation across multiple data stores while maintaining fast query response times
Solution Approach 2:
The system divides complex federated queries into smaller sub-queries that can be executed independently on individual data stores. Each data store processes only its relevant portion, reducing the overall resource overhead compared to scanning all data stores simultaneously, while still delivering complete query results
2Adaptability or versatility
If federated querying across multiple data stores with different structures is implemented, then data accessibility is improved, but system complexity increases
Solution Approach 1:
The system introduces materialized views as intermediary structures between diverse source data stores and user queries. These materialized views provide a unified, standardized representation of data from multiple sources with different structures, allowing queries to be executed without directly handling the complexity of underlying data store heterogeneity
Solution Approach 2:
The system creates a universal querying interface that can access multiple different data stores through a common materialized view structure. This single interface handles diverse data sources with different architectures and structures, providing universal data accessibility without requiring separate handling logic for each data store type
3Quantity of substance
If data volume increases to accommodate growing customer base, then service capacity is improved, but search efficiency decreases
Solution Approach 1:
The system pre-computes and stores materialized views of data before they are needed for queries. As data volume grows, these pre-computed results remain readily available for fast retrieval, preventing search efficiency from deteriorating despite increased data quantities
Solution Approach 2:
The system creates copied representations of source data in materialized views that are optimized for querying. These copies contain pre-processed and organized data that can be searched efficiently without requiring full scans of the original large-volume source data stores
Data Source
AI summary
Systems, methods, and computer-readable media for asynchronous (async) querying are described. In embodiments, a system may convert a user-issued query into a distributed execution instruction set (DEIS), and issue the DEIS to data stores that may have various database structures. The system may obtain database objects from the data stores, and store a result set indicating the obtained database objects in a location specified by the user-issued query. The system may also provide mechanisms to allow users to view progress of their async query jobs and/or cancel ongoing async query jobs. Other embodiments may be described and/or claimed.