Estimate-Based Scheduler for Distributed Storage Task Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies in processing redundant data due to random task distribution methods, leading to resource wastage, higher costs, and poor performance, especially as demand for real-time processing increases.
Innovation Solution
Implementing an estimate-based scheduler that receives processing data for storage nodes with redundant data chunks, determines task time estimates for each data path, and selects optimal paths for compute tasks to meet processing time thresholds, while also splitting tasks if necessary and adjusting estimates based on actual performance feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If random selection method is used to assign compute tasks to storage nodes, then task distribution is simple to implement, but processing efficiency deteriorates due to node failures, latency, and load imbalance
Solution Approach 1:
The system implements a feedback mechanism where the scheduler receives actual processing time data from completed tasks and uses this information to update estimated processing times for future task assignments. This closed-loop feedback system allows the scheduler to learn from past performance and continuously improve task distribution efficiency, resolving the contradiction between simple random assignment and efficient optimized assignment.
Solution Approach 2:
The scheduler performs preliminary actions by pre-calculating estimated processing times for each storage node before assigning compute tasks. This advance preparation allows the system to make informed decisions about task distribution without requiring complex real-time calculations during task assignment, maintaining simplicity while improving efficiency.
2Reliability
If more redundant copies of data chunks are distributed across storage nodes, then system reliability improves, but resource wastage increases
Solution Approach 1:
The system dynamically changes the parameter of redundancy level by adjusting the number of replicate tasks assigned to different storage nodes based on their estimated processing times and current load conditions. Instead of maintaining uniform high redundancy across all nodes, the system optimizes redundancy distribution to match actual node performance characteristics, improving reliability where needed while reducing resource wastage where unnecessary.
3Productivity
If compute tasks are distributed across multiple data nodes, then processing parallelism increases, but coordination complexity and communication overhead increase
Solution Approach 1:
The scheduler acts as an intermediary between the task submission system and the distributed storage nodes. It consolidates task assignment decisions, manages coordination logic, and handles communication overhead centrally, allowing individual storage nodes to execute tasks independently without complex peer-to-peer coordination. This mediator approach enables high parallelism while containing coordination complexity in a single component.
4Manufacturing precision
If task processing time threshold is set strictly, then service level requirements are met, but task completion rate decreases due to premature task rejection
Solution Approach 1:
The system applies partial action by selectively enforcing the processing time threshold only for tasks where it is critical to meet service level requirements. For tasks where the threshold would prevent completion without significant benefit, the system allows flexibility, maintaining completion rates while still meeting essential service level agreements through targeted threshold enforcement.
Data Source
AI summary
Systems and methods for distributed storage and processing systems using estimate-based schedulers are described. A node receives estimated processing data for each storage device including redundant copies of data chunks for a data unit. The node determines, based on the estimated processing data and data paths to each data chunk, a task time estimate for data paths to each data chunk and selects data paths for at least one copy of each data chunk to be processed using a corresponding set of compute tasks. The compute tasks are sent for processing based on the assignments of the node.