Distributed Map Reduce Node Selection Cost Deadline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed map reduce networks face challenges in efficiently managing large data processing jobs due to variability in node capabilities and geographical distribution, leading to difficulties in meeting time deadlines while optimizing cost and resource utilization.
Innovation Solution
A distributed map reduce network system that utilizes an orchestrator to select nodes based on cost and geographical location, forming sub-clusters for failover and optimizing job assignment, ensuring timely completion of map reduce jobs by replacing nodes that cannot meet deadlines with cheaper alternatives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If nodes are selected based on lowest cost value, then cost is minimized, but job completion time may exceed deadline
Solution Approach 1:
The system dynamically changes the selection parameters by considering both cost value and geographical location metrics. When initial low-cost node selections fail to meet deadlines, the system adjusts by incorporating location-based filtering to select from alternative nodes that can complete jobs within required timeframes.
Solution Approach 2:
The node selection process is dynamic rather than static. The system continuously evaluates both cost and location parameters, and adapts its selection criteria based on job requirements and deadline constraints. The orchestrator can replace nodes during execution if deadlines are not being met.
2Reliability
If nodes are selected based on geographical location for failover, then reliability is improved, but cost increases
Solution Approach 1:
The system applies different selection criteria to different roles within the distributed system. Primary nodes are selected based on lowest cost, while backup/failover nodes are selected based on geographical proximity to ensure reliability. This local differentiation of selection quality resolves the contradiction between cost and reliability.
Solution Approach 2:
The node selection is segmented into different functional categories: primary execution nodes selected for cost efficiency, and backup nodes selected for geographical proximity. This segmentation allows the system to optimize for different parameters in different roles, achieving both cost efficiency and reliability.
3Productivity
If more nodes are used for parallel processing, then productivity increases, but cost increases
Solution Approach 1:
The system uses partial parallelization rather than maximizing node count. It selects the minimum number of nodes required to meet the deadline, using geographical location information to determine optimal node distribution. This avoids excessive node deployment while still achieving necessary productivity.
Data Source
AI summary
Embodiments relate to a distributed map reduce network. An aspect includes receiving a description of a map reduce job comprising a time deadline. Another aspect includes selecting a set of candidate nodes from a plurality of nodes of the distributed map reduce network, wherein the set of candidate nodes are selected based on cost value. Another aspect includes determining an amount of time required to perform the map reduce job by the set of candidate nodes. Another aspect includes, based on the determined amount of time not meeting the time deadline: selecting a particular node of the set of candidate nodes for replacement based on the geographical location of the particular node; selecting a next node of the plurality of nodes based on the cost value of the next node; and replacing the particular node in the set of candidate nodes with the next node.