Cost-Based Optimizer for Cross-Platform Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches to cross-platform data processing are expensive, inflexible, and error-prone due to the need for ad-hoc programming across diverse data processing platforms, which complicates data query and processing tasks, especially when datasets reside on multiple platforms in different formats.
Innovation Solution
A cost-based optimizer that generates and selects execution plans across multiple data processing platforms to minimize total execution cost, considering both the cost of executing tasks and data movement between platforms, while accommodating different operations and processing abstractions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If ad-hoc programming is used to connect multiple data processing platforms, then cross-platform data processing can be achieved, but the system becomes expensive, inflexible, and error-prone
Solution Approach 1:
The patent introduces a query translation service as an intermediary component that sits between the query interface and multiple data processing platforms. This service automatically translates queries across different platforms without requiring ad-hoc programming for each platform connection, thereby reducing programming complexity while maintaining cross-platform capability.
Solution Approach 2:
The query translation service provides a universal interface that can handle queries across multiple different data processing platforms (Hadoop, Spark, Presto, etc.) through a single unified service, eliminating the need for separate programming approaches for each platform and reducing overall system complexity.
2Productivity
If multiple specialized data processing platforms are used to handle diverse datasets, then functionality and performance are improved, but data query and processing tasks become highly complex due to multiple formats
Solution Approach 1:
The query translation service acts as a mediator that handles the complexity of querying data in multiple formats across different platforms. It automatically translates and adapts queries to work with various data formats and platform-specific protocols, thereby maintaining high data processing performance while reducing query complexity for users.
Solution Approach 2:
The system segments the complexity handling into distinct layers: the query translation service handles format conversion and platform-specific syntax, while the underlying data processing platforms handle their native operations. This segmentation allows each component to specialize without exposing the full complexity to users.
3Adaptability or versatility
If data resides on multiple platforms in multiple formats, then diverse data analytics can be performed, but the cost of moving data across platforms increases
Solution Approach 1:
The query translation service serves as an intermediary that optimizes data movement across platforms by intelligently routing queries to the appropriate platform based on data location and format. This reduces unnecessary data movement and associated costs while maintaining the ability to perform diverse data analytics across multiple platforms.
Data Source
AI summary
The present disclosure generally relates to a cost-based optimizer for efficiently processing data through the use of multiple different data processing platforms. The cost-based optimizer may receive an input plan for processing data that includes a number of base operators. The cost-based optimizer may then determine execution operators for each base operator, where each execution operator corresponds to a different data processing platform. From the execution operators, the cost-based optimizer may determine possible subplans for executing the input plan on one or more data processing platforms. The cost-based optimizer may determine the cost of executing each possible subplan and choose the subplan with the lowest cost.