Map Reduce Scheduler Service for Spark Job Orchestration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data platforms face challenges in efficiently executing large-scale data processing jobs, such as Spark jobs, while optimizing costs and developer productivity.
Innovation Solution
The Map/Reduce platform addresses these challenges by incorporating a Scheduler Service (MRSS) that abstracts the complexity of running Spark jobs and a Controller that manages and orchestrates Spark clusters, utilizing spot instances and optimizing cluster scaling for cost and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a centralized scheduling service is implemented to manage Spark jobs, then job submission and monitoring become simplified for developers, but system complexity increases due to the need for centralized coordination and cluster management
Solution Approach 1:
The patent introduces a centralized Scheduler Service (MRSS) that acts as an intermediary between developers and the distributed Spark cluster. The MRSS receives job requests via HTTP API, manages job queues, coordinates resource allocation, and provides centralized monitoring. This mediator abstracts the complexity of distributed cluster management from developers while maintaining system-wide coordination, thus simplifying operations without requiring developers to directly manage cluster complexity
2Loss of energy
If spot instances are utilized to reduce costs, then computational expense decreases, but system reliability deteriorates due to the preemptible nature of spot instances
Solution Approach 1:
The patent implements dynamic cluster management where the Controller continuously monitors spot instance availability and adjusts cluster composition in real-time. When spot instances are preempted, the system dynamically replaces them with on-demand instances or other available spot instances. This dynamic adaptation allows the system to maximize cost savings from spot instances while maintaining job execution reliability through automatic instance replacement and cluster rescaling
Solution Approach 2:
The system implements preemptive measures by maintaining a pool of reserved on-demand instances as a cushion against spot instance preemption. When spot instances are at risk of preemption or actually preempted, the system can quickly transition to the reserved capacity. This prior cushioning ensures that critical jobs maintain reliability while still benefiting from cost savings, as the system has pre-prepared backup capacity ready for immediate deployment
3Reliability
If multiple small clusters are used instead of few large clusters, then robustness improves through distributed failure isolation, but system complexity increases due to managing multiple cluster instances
Solution Approach 1:
The patent segments the processing workload across multiple independent small Spark clusters rather than using a single large cluster. Each small cluster is independently managed and can fail without affecting others, providing failure isolation and improved robustness. The Scheduler Service manages the portfolio of small clusters, allocating jobs across them based on availability and capacity. This segmentation approach distributes risk while the centralized scheduler abstracts the complexity of managing multiple cluster instances
4Productivity
If automatic cluster scaling is implemented to optimize resource utilization, then productivity increases through efficient resource allocation, but system complexity increases due to dynamic provisioning requirements
Solution Approach 1:
The patent implements automatic cluster scaling through a feedback mechanism where the Controller continuously monitors cluster utilization metrics (CPU usage, memory consumption, job queue depth, pending tasks) and dynamically adjusts cluster capacity accordingly. When utilization exceeds thresholds, the system automatically provisions additional instances; when utilization is low, it deallocates resources. This closed-loop feedback control optimizes resource utilization efficiency while the automated nature of the process abstracts the complexity of dynamic provisioning from manual operations
Data Source
AI summary
The inventions related to Map/Reduce, a big data platform that helps in running large scale data processing jobs, e.g. spark jobs, in a batch pipeline used for building an Identity Graph and other data products.