ML-Based Data Analytics Engine Configuration for Executor Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for configuring data analytics engines are inefficient and time-consuming, requiring manual trial-and-error and deep knowledge of hardware parameters, and lack effective use of Supervised Machine Learning (ML) for optimizing configuration parameters.
Innovation Solution
A method and system using Supervised ML to predict optimal configuration parameters for data analytics engines by analyzing input parameters and deploying models like SARIMAX and Regression to determine the number of executor instances, cores, memory, and parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual trial-and-error configuration is used, then configuration accuracy can be achieved, but time consumption and operational complexity increase significantly
Solution Approach 1:
The system enables self-service configuration by automatically determining optimal executor parameters through machine learning models. The ML model analyzes historical job performance data and system state to autonomously configure executor instances, cores, memory, and parallelism without requiring manual trial-and-error by operators.
Solution Approach 2:
The patent replaces the mechanical manual configuration process with an automated machine learning-based system. The ML model substitutes human operators in the configuration task, using algorithms to predict optimal parameters based on input features such as data size, job type, and cluster characteristics.
2Reliability
If exhaustive searching is used to optimize hardware settings, then optimal configuration can be found, but computational cost and time effort increase hugely
Solution Approach 1:
The system performs preliminary action by pre-training machine learning models on historical configuration data and job performance metrics. This pre-computed knowledge is stored and reused for future configuration tasks, avoiding the need to perform exhaustive searches for each new job. The model learns from past experiences to make rapid predictions without repeating costly optimization processes.
Solution Approach 2:
The patent transforms the configuration optimization problem from an exhaustive parameter search into a machine learning prediction task. By changing the approach from direct optimization to predictive modeling, the system achieves high-quality configurations with significantly reduced computational cost. The ML model learns complex parameter relationships from training data and generalizes to new scenarios.
3Ease of operation
If static configuration method is used, then simplicity is maintained, but adaptability to different job requirements deteriorates
Solution Approach 1:
The system introduces dynamics by using machine learning models that adapt configuration recommendations based on varying job requirements and system states. Unlike static configuration methods, the ML model dynamically adjusts executor parameters according to the specific characteristics of each job, data size, and cluster conditions, maintaining both simplicity and adaptability.
Solution Approach 2:
The patent enables parameter changes by training the ML model on diverse configuration scenarios and job types. The model learns to adjust executor instances, cores, memory, and parallelism parameters based on input features, providing adaptive recommendations without requiring complex manual configuration. This maintains ease of operation while achieving high adaptability.
4Speed
If more executor instances are allocated, then processing speed improves, but resource utilization efficiency may deteriorate
Solution Approach 1:
The machine learning model optimizes the executor instances parameter by analyzing the relationship between data size, job complexity, and processing speed. The model predicts the optimal number of executor instances that achieves high processing speed while avoiding over-provisioning that would waste resources. This balanced configuration maximizes resource utilization efficiency.
Data Source
AI summary
The invention relates to a method and system for predicting optimal configuration parameters for a data analytics engine. The method and system utilizes Supervised Machine Learning (ML) techniques for predicting optimal configuration parameters for the data analytics engine to run a specific application in a given time frame, wherein the optimal configuration includes estimating an optimal number of executor instances in real-time. The invention analyzes input parameters such as data metrics, software metrics, and hardware metrics to determine if the input parameters comprise a new dataset or a continuous dataset and deploy one or more models such as a Forecasting Model and a Regression Model. The one or more models then derive and allocate the possible number of executor instances. The executor instances allocated can be a number of executors, executor cores, the memory of executors, driver memory, and parallelism.