Spark Performance Prediction via Log Parsing and Modular Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for predicting application performance on Spark platforms are inadequate, as they often rely on black box approaches that fail to account for the various configurable parameters of big data systems, leading to inaccurate performance predictions.
Innovation Solution
A system and method that predicts application performance on Spark platforms by executing applications on a small Spark cluster with small data, parsing logs to collect granular performance data, estimating scheduler delay, shuffle operation time, and JVM overheads, and simulating stage execution to estimate execution times, thereby facilitating accurate performance prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If black box machine learning approaches are used for performance prediction, then prediction capability is provided, but prediction accuracy deteriorates due to inability to account for configurable parameters
Solution Approach 1:
The performance prediction system is segmented into multiple independent modules: log parser module for collecting granular performance data, scheduler delay predictor module for estimating scheduler delays, shuffle time predictor module for estimating shuffle operation times, and execution time predictor module for estimating JVM execution times. Each module handles a specific aspect of performance prediction, improving overall accuracy while maintaining system transparency through modular design.
Solution Approach 2:
The system introduces an intermediary log parser module that bridges the gap between black box machine learning approaches and system transparency. This module collects granular performance data from application logs, making the underlying system behavior observable and analyzable, thereby improving prediction accuracy without sacrificing transparency.
2Measurement precision
If performance testing is done on production systems with large data size, then accurate performance data is obtained, but development agility deteriorates due to resource constraints and risk
Solution Approach 1:
The system performs preliminary performance testing on small Spark clusters with small data sizes before production deployment. The log parser module collects performance data from these preliminary runs, and the predictor modules estimate performance metrics that can be extrapolated to predict production performance, enabling early performance validation without consuming production resources.
Solution Approach 2:
The system creates a simplified copy of the production environment using small Spark clusters for performance testing. This copy replicates the essential characteristics of the production system while using minimal resources, allowing developers to perform comprehensive performance testing and validation without impacting production systems or sacrificing development agility.
3Productivity
If small Spark clusters with small data are used for performance prediction, then development cost and time are reduced, but prediction accuracy may deteriorate due to scale differences
Solution Approach 1:
The system changes the parameters of performance measurement by collecting granular performance data at the task level rather than aggregate levels. The log parser module extracts detailed metrics including scheduler delays, shuffle operation times, and JVM execution times for individual tasks, enabling accurate prediction models that can scale from small cluster results to large cluster predictions through parameter extrapolation.
Solution Approach 2:
The system employs dynamic prediction models that adapt to different cluster sizes and configurations. The scheduler delay predictor, shuffle time predictor, and execution time predictor modules use the collected granular data to dynamically estimate performance metrics for various scale scenarios, allowing accurate prediction of large cluster performance based on small cluster experiments.
Data Source
Figure 1
Figure 2(a)
Figure 2(b)
AI summary
A system and method for estimating execution time of an application with Spark platform in a production environment. The application on Spark platform is executed as a sequence of Spark jobs. Each Spark job is executed as a directed acyclic graph (DAG) consisting of stages. Each stage has multiple executors running in parallel and the each executor has set of concurrent tasks. Each executor spawns multiple threads, one for each task. All jobs in the same executor share the same JVM memory. The execution time for each Spark job is predicted as summation of the estimated execution time of all its stages. The execution time constitutes scheduler delay, serialization time, de-serialization time, and JVM overheads. The JVM time estimation depends on type of computation hardware system and number of threads.