Task Scheduling Simulation for Mixed Batch and Online Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Yarn simulation system (Scheduler Load Simulator, SLS) in the big data processing system Hadoop can only simulate Map-Reduce single batch tasks, failing to account for mixed scheduling and resource utilization in cloud computing data centers where both batch and online tasks are run, and lacks accurate representation of CPU and memory resources.
Innovation Solution
A task scheduling simulation system is developed, comprising a data preprocessing subsystem and a task scheduling subsystem. The data preprocessing subsystem filters and processes cloud computing log information to extract task details, while the task scheduling subsystem manages task queues and updates resource utilization to simulate the mixed scheduling of batch and online tasks, reflecting the actual production environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the Yarn simulation system (SLS) is used to simulate batch tasks, then the simulation of Map-Reduce single batch tasks is achieved, but the mixed scheduling of batch and online tasks cannot be simulated
Solution Approach 1:
The patent extends the simulation system to handle multiple task types (batch tasks and online tasks) simultaneously through a unified simulation framework. The system now supports mixed scheduling scenarios by integrating online task simulation capabilities alongside existing batch task simulation, making the system versatile for various cloud computing scheduling scenarios without requiring separate simulation systems
2Measurement precision
If the simulation system focuses on single task type, then the system design is simple, but the resource utilization representation is inaccurate for cloud computing environments
Solution Approach 1:
The patent segments the simulation system into distinct modules: batch task simulation module, online task simulation module, and resource utilization calculation module. Each module handles specific aspects of task simulation independently, allowing the system to accurately represent complex resource utilization patterns while maintaining manageable system structure through clear separation of concerns
3Reliability
If production environment log records are used, then real cloud computing scenarios are captured, but abnormal data and incomplete information affect simulation accuracy
Solution Approach 1:
The patent implements preliminary data preprocessing and filtering mechanisms that clean and validate production environment log records before they are used in simulations. The system pre-processes raw logs to remove abnormal data, fill missing information, and standardize formats, ensuring high-quality input data for accurate simulation results while maintaining the authenticity of real cloud computing scenarios
Data Source
AI summary
The application provides a task scheduling simulation system, comprising a data preprocessing subsystem and a task scheduling subsystem. The data preprocessing subsystem filters the input cloud computing log information for abnormal data and extracts the running time of each task. The task scheduling subsystem enqueues or dequeues tasks from the batch task and real-time task running queues of each node, and keeps the tasks currently running in the cluster consistent with the actual production environment, and updates the number of CPU cores and the used and available memory capacity of each node according to resource requirement of each task. The mixed scheduling simulation of batch tasks and online tasks can be realized, and the resource simulation of the heterogeneous CPU core number and memory capacity of the cluster nodes can be simulated.