External Table Function Parallelism Configuration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cloud database systems lack the ability to call external remote software components for providing functionality, leading to limitations in data analysis and processing, especially with transient or frequently changing data sources, resulting in high latency and data unusability due to offline export and import mechanisms.
Innovation Solution
A cloud database system that enables users to define and invoke external functionality by binding SQL functions, table functions, or procedures to remote software code exposed as a web API, allowing queries against external data sources using a table-like abstraction, with temporary security credentials managed by a cloud computing platform for secure authentication and data exchange.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional cloud database systems use offline export and import mechanisms for external data sources, then data can be processed, but latency increases and data becomes unusable when frequently changing
Solution Approach 1:
The system transitions from static offline export/import mechanisms to dynamic real-time data exchange. External table functions enable continuous querying of external data sources, allowing the system to adapt to frequently changing data without requiring periodic re-export and re-import operations.
Solution Approach 2:
External table functions serve as an intermediary layer between the cloud database system and external data sources. This mediator enables direct querying of external data through SQL interfaces, eliminating the need for intermediate export/import steps and reducing data exchange latency.
2Adaptability or versatility
If cloud database systems integrate external functionality through remote software components, then data analysis capability improves, but system complexity increases
Solution Approach 1:
External table functions provide a universal interface for accessing diverse external data sources. By using a single SQL-based table function mechanism, the system can access multiple external sources (databases, APIs, files) without requiring separate integration logic for each source type, thus improving versatility while managing complexity.
Solution Approach 2:
The external table function mechanism enables self-service integration where users can define and query external data sources using standard SQL syntax without requiring complex configuration or deep understanding of the underlying remote software components. The system automatically handles the complexity of remote communication and data format conversion.
3Productivity
If external table functions are invoked with high parallelism, then query processing speed increases, but resource consumption increases
Solution Approach 1:
The system implements dynamic parallelism where the degree of parallelization is adjusted based on query characteristics and available resources. Rather than always using maximum parallelism, the system applies partial parallelism appropriate to each query, avoiding excessive resource consumption while maintaining high throughput where beneficial.
Solution Approach 2:
The system dynamically changes parallelism parameters during query execution based on feedback from the external data source response times and available compute resources. This allows the system to optimize the balance between query processing speed and resource consumption by adjusting the number of parallel workers allocated to external table function invocations.
Data Source
AI summary
A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.