Heterogeneous Big Data Application Generation via Execution Dependency Graph
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Big Data processing approaches require developers to tailor solutions for each specific platform, leading to increased development time and complexity due to differences in data partitioning, shuffling, and aggregation capabilities.
Innovation Solution
A method that creates a development architecture capable of addressing heterogeneous application platforms by specifying computation and communication patterns independently of the target architecture, allowing for optimal concurrency and efficient task mapping across clusters, thereby reducing development effort and improving maintainability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If developers tailor solutions for each specific platform, then platform-specific optimization is achieved, but development time and complexity increase
Solution Approach 1:
The patent introduces a unified application creation method that can generate applications for multiple heterogeneous Big Data platforms (Spark, Hadoop, UIMA-AS) using the same input specifications. The system translates high-level computation patterns into platform-specific implementations automatically, eliminating the need for developers to rewrite applications for each platform while maintaining platform-specific optimizations.
2Productivity
If developers learn and specifically develop for each platform, then platform-specific optimization is achieved, but development time increases
Solution Approach 1:
The system pre-establishes translation rules and templates that map high-level computation patterns to multiple platform implementations. When a developer specifies computation patterns, the system automatically translates them into platform-specific code before execution, eliminating the time-consuming process of learning and adapting to different platform APIs and data partitioning mechanisms.
3Productivity
If applications are rewritten to optimize data partitioning and shuffling, then computational efficiency improves, but development complexity increases
Solution Approach 1:
The patent introduces an intermediary translation layer that sits between the developer's high-level computation specifications and the platform-specific data partitioning mechanisms. This intermediary automatically handles the complex translation of computation patterns into optimized data partitioning, shuffling, and aggregation operations for each platform, shielding developers from the underlying complexity while maintaining computational efficiency.
Data Source
AI summary
This invention relates to a method and system for creating Big Data applications that can be executed on heterogeneous clusters. The applications can be executed on a particular platform, such as SPARK or UIMA-AS, but the method and system are able to translate the input to these targeted platforms without the developer needing to tailor the application specifically to the platform. The method and system are based on the use of an execution dependency graph, a cluster configuration, and a data size to create a stages table. The stages table is then optimized to increase the overall efficiency of the heterogeneous cluster. The stages table is then translated into a platform specific Big Data application.