Transcompiler for Automatic MapReduce Job Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software programs are not easily scalable to handle large datasets, as they often require significant expertise to adapt to distributed computing frameworks like MapReduce, leading to inefficiencies and high costs in porting existing software to distributed systems.
Innovation Solution
A transcompiler is used to identify loops in non-distributed source code, generate MapReduce jobs, and optimize them using machine learning algorithms, enabling automatic porting of data-driven programming languages to distributed frameworks without the need for manual expertise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing software programs are ported to distributed computing frameworks manually, then the software can process large datasets, but it requires significant developer expertise and time, increasing cost and complexity
Solution Approach 1:
The system automatically analyzes non-distributed source code and generates distributed MapReduce jobs without requiring manual intervention from developers. The transcompiler self-services by parsing the original code, identifying loops and data processing logic, and automatically generating equivalent distributed tasks, thereby eliminating the time and expertise loss associated with manual porting.
Solution Approach 2:
A transcompiler serves as an intermediary between non-distributed source code and distributed MapReduce frameworks. This intermediary tool translates the original programming language into MapReduce jobs, handling the complexity of conversion and enabling existing software to run on distributed systems without direct developer involvement in the translation process.
2Adaptability or versatility
If software is rewritten using MapReduce, then scalability is achieved, but it requires specialized developer knowledge and increases the risk of incorrect implementation
Solution Approach 1:
The transcompiler automatically determines mapper keys and generates MapReduce jobs by analyzing the original source code's data processing logic. This self-service approach eliminates human error in key selection and ensures reliable translation while maintaining scalability, as the system independently handles the complex task of generating correct distributed tasks from non-distributed code.
Solution Approach 2:
The manual mechanical process of selecting mapper keys and writing MapReduce code is replaced by an automated computational system. The transcompiler uses algorithmic analysis to substitute human decision-making with automated code generation, ensuring consistent and error-free translation while preserving the scalability benefits of MapReduce.
3Productivity
If a transcompiler automatically generates distributed software packages, then porting speed increases, but the complexity of the compilation process increases
Solution Approach 1:
The transcompiler divides the complex analysis process into segmented steps: parsing source code, identifying loops, determining data processing logic, and generating MapReduce jobs. This segmentation manages complexity by breaking down the translation task into manageable phases, each handled by specific processing modules within the transcompiler system.
Solution Approach 2:
The transcompiler acts as an intermediary layer that encapsulates the complexity of distributed framework integration. By placing the complexity within this intermediate translation layer, the original non-distributed code remains simple and unchanged, while the transcompiler handles all the complex analysis and generation tasks internally, producing simplified MapReduce output.
4Loss of time
If existing software is not rewritten for distributed computing, then development time is maintained, but processing speed for large datasets decreases
Solution Approach 1:
The transcompiler creates a copy of the original non-distributed software's processing logic and translates it into the distributed MapReduce paradigm. This copying approach preserves the original functionality and development time investment while transforming the execution model to achieve parallel processing speedups on large datasets, effectively decoupling development time from processing performance.
Data Source
AI summary
Systems and methods are provided for transcompiling non-distributed source code for a non-distributed software program into a distributed software package for implementation on a distributed computing system. A transcompiler can identify loops within non-distributed source code written in a data-driven language. The transcompiler can generate MapReduce jobs using mapper keys based on grouping indicators associated with each of the loops. The MapReduce jobs can be linked together based on input-output connections of the loops in the non-distributed source code. Then, the transcompiler can generate a distributed software package including the generated MapReduce jobs to implement the same functionality as the non-distributed source code on the distributed computing system, thereby improving the speed of execution over very large datasets. The distributed software package can be optimized using machine learning searching algorithms. The distributed software package can also be optimized based on execution usage statistics.