Polymorphic SQL Functions Parallel Execution Plan
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data processing frameworks face limitations in combining executions of different analytic computations, leading to high unnecessary input/output costs and poor performance due to unoptimized data movement and restricted expression of analytic functions in query languages like SQL, which restricts their connectivity to business applications.
Innovation Solution
The implementation of a parallel architecture that integrates traditional transaction user-defined functions with MapReduce and graph computation models, enabling flexible SQL expressions for user-defined scalar and aggregate functions, and a global planner to optimize data movements, allowing for the composition of analytic functions across different programming paradigms like Java and C, and extending SQL capabilities to connect with business applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional big data processing frameworks are used, then data processing can be performed, but unnecessary I/O costs increase and performance deteriorates due to unoptimized data movement
Solution Approach 1:
The system performs preliminary actions by generating an optimized execution plan before data processing begins. The global planner analyzes the query and creates a data movement optimization plan that prepares the optimal execution strategy in advance, preventing unnecessary I/O operations during actual data processing.
Solution Approach 2:
The execution plan is dynamically optimized based on the specific query and data distribution. The system adapts the data movement strategy to match the actual processing requirements, allowing flexible adjustment of execution paths to minimize I/O costs while maintaining processing efficiency.
2Adaptability or versatility
If SQL query language is used, then business applications can connect to databases, but expression of analytic functions is restricted
Solution Approach 1:
The system introduces an intermediary layer (global planner and execution plan generator) between the SQL query language and the distributed processing framework. This intermediary translates standard SQL expressions into optimized distributed execution plans, enabling complex analytic functions to be expressed using familiar SQL syntax while the system handles the complexity of distributed execution.
3Power
If data is processed in a distributed environment, then processing capacity increases, but data transfer between nodes creates performance bottlenecks
Solution Approach 1:
The system segments the distributed processing into optimized stages based on data locality and computation requirements. By dividing the execution plan into segments that minimize cross-node data transfer and maximize local processing, the system maintains high processing capacity while reducing network bottlenecks.
Data Source
AI summary
Disclosed are systems and methods for implementing contract-based polymorphic and parallelizable user-defined scalar and aggregate functions. The systems and methods can include receiving a query including a plurality of user-defined functions, parsing the query into a plurality of nodes (e.g., basic operation unit or atomic operator), generating an execution plan that minimizes data transfer between the plurality of nodes, and executing the plan in a distributed environment. Each of the plurality of user-defined functions can correspond to one of a plurality of nodes.