Automated MapReduce Job Configuration for Parallel Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business analysts and non-computer scientists face challenges in performing fine-grained analyses of large data sets using MapReduce due to the high skill level required for manipulating massive parallel computing architectures and the inability to properly allocate data among nodes for optimizations, leading to inefficient computation resources and transient data analysis needs.
Innovation Solution
A process that involves parsing data analysis specifications, determining implicated data, distributing data based on an index to keep related values on the same nodes, sending relevant rules to corresponding nodes, and aggregating results, while composing mapper and reducer programs to automate data analysis tasks, thereby simplifying the use of distributed parallel algorithms like MapReduce.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If MapReduce is used for distributed parallel data analysis, then computing power and processing capability are improved, but the skill level required and system complexity increase
Solution Approach 1:
The patent introduces an intermediary system that automatically generates MapReduce job configurations from high-level data analysis specifications. This intermediary layer translates business requirements into technical implementation details, shielding users from MapReduce complexity while maintaining access to distributed computing power.
Solution Approach 2:
The system provides a universal interface that handles multiple data analysis tasks through a single automated configuration system. Instead of requiring users to learn different MapReduce implementations for various analysis types, the system offers a unified approach that works across diverse analytical scenarios.
2Measurement precision
If manual data allocation among nodes is performed, then data analysis precision and optimization are improved, but time consumption and operational complexity increase
Solution Approach 1:
The system performs preliminary automatic data allocation and job configuration before execution. By pre-determining optimal data distribution across nodes based on the analysis specification, the system eliminates manual allocation time while ensuring precision is maintained through automated optimization.
Solution Approach 2:
The system enables self-service automated configuration where the job management system automatically determines data allocation strategies, generates mapper and reducer programs, and configures execution parameters without requiring manual intervention. This maintains optimization quality while eliminating manual configuration time.
3Ease of operation
If automated data allocation is implemented, then ease of operation is improved, but control over data distribution may be reduced
Solution Approach 1:
The system incorporates feedback mechanisms where execution results and performance metrics are analyzed to automatically adjust and optimize data allocation strategies. This feedback loop maintains control flexibility by allowing the system to adapt to different scenarios while keeping the user interface simple and automated.
Data Source
AI summary
Provided is a process having steps including obtaining a specification of a data analysis to be performed in parallel on a computing cluster; parsing the specification of the data analysis; determining which data is implicated in each portion of the data analysis to be assigned to a plurality of computing nodes of the computing cluster; determining that a portion of the implicated data is not already present in memory of at least some of the plurality of computing nodes of the computing cluster; distributing the portion of the implicated data according to an index that positions related values of the data on the same computing nodes of the computing cluster; determining which computing nodes of the computing cluster have data relevant to which rules in the data analysis and send relevant rules to the corresponding computing nodes; executing the rules on the computing nodes; and aggregating results of executing the rules.