Parallelized External Table Access via Work Granule Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Database Management Systems (DBMS) face limitations in achieving parallelism when executing queries that access external tables, as existing execution plans do not include work granules for generating rows from external data sources that can be executed in parallel with other granules, thereby hampering the degree of parallelism and query performance.
Innovation Solution
The approach involves generating execution plans with multiple work granules that access an external data source, allowing each granule to be assigned to a slave process for parallel execution, utilizing a distributed data access system where data sources are accessible across a cluster of data nodes, and specifying the appropriate code modules to generate rows from heterogeneous data formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If execution plans include work granules for accessing external data sources, then query performance is improved, but the degree of parallelism is hampered because external table row generation cannot be divided into multiple parallel work granules
Solution Approach 1:
The patent segments external table access into multiple work granules by dividing the external data source into partitions. Each work granule is assigned to a different slave process and accesses a specific partition of the external data source, enabling parallel execution. This segmentation resolves the contradiction by allowing external table access to be divided into independent parallel tasks while maintaining manageable complexity through structured partitioning.
2Loss of time
If multiple slave processes execute work granules in parallel, then query execution time is reduced, but external table access cannot participate in parallel execution with other granules
Solution Approach 1:
The external data source is segmented into multiple partitions, with each partition accessible by a separate work granule. This allows multiple slave processes to execute external table access operations in parallel simultaneously with other work granules, reducing query execution time while maintaining adaptability through the standardized partitioned access interface.
Solution Approach 2:
The patent creates a universal work granule structure that can handle both internal and external table access uniformly. The external table access work granules follow the same execution model as internal table access, enabling them to participate in the same parallel execution framework and be scheduled alongside other granules without requiring special handling.
3Device complexity
If external table access is implemented as a single work granule, then implementation complexity is reduced, but the degree of parallelism is limited
Solution Approach 1:
The patent implements a segmented approach where the external data source is divided into partitions, and each partition is accessed by a separate work granule. This segmentation enables parallelism while keeping implementation complexity manageable through the use of standardized partitioning logic and uniform work granule structures that can be generated automatically by the execution plan compiler.
Data Source
AI summary
An approach, referred to herein as parallelized-external-table access, generates rows from a single external table in parallel for a given query. Under parallelized-external-table access, an execution plan generated for the query includes multiple work granules that generate rows for a single external table from a data source. Such work granules are referred to herein as external work granules. Each external work granule of the execution plan may be assigned to a slave process, which executes the external work granule in parallel with another slave process executing another external work granule. External tables are accessible on a cluster of data nodes in a distributed data access system (e.g. Hadoop Distributed File System) connected to a DBMS.