Continuous Query Scheduling and Splitting in Cluster Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cluster-based data storage systems like ECS face destabilizing load spikes due to high resource intensity of continuous query tasks, especially when limited system resources are reserved for monitoring, leading to high latency and workload issues.
Innovation Solution
Implementing continuous query task scheduling and splitting techniques, where tasks are staggered with offset times and divided into subtasks based on load leveling and cardinality, to distribute the load and prevent spikes, using a monitoring subsystem to manage scheduling and splitting without overwhelming the system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If continuous query tasks are executed with a longer period to reduce frequency, then the number of executions is reduced, but the amount of data to read and process increases, causing higher resource consumption and load spikes
Solution Approach 1:
The patent divides a single continuous query task into multiple subtasks that process different portions of the data range. Each subtask handles a specific segment (e.g., different time windows or data partitions), reducing the processing load per task while maintaining comprehensive monitoring coverage. This segmentation allows parallel execution of lighter subtasks instead of one heavy query.
Solution Approach 2:
The patent introduces a new dimension of parallelism by executing multiple continuous query tasks simultaneously with different offset times. Instead of running one query at a time sequentially, the system launches multiple queries that operate on different data segments in parallel, distributing the overall workload across multiple processing threads and reducing the burden on any single resource.
2Reliability
If continuous query tasks are executed frequently to ensure real-time monitoring, then monitoring accuracy is improved, but load spikes destabilize the monitoring subsystem
Solution Approach 1:
The patent implements periodic execution of continuous query tasks with staggered offset times. Each query task runs at regular intervals but with different offsets (e.g., Task 1 at 00:00, Task 2 at 06:00, Task 3 at 12:00), creating a distributed periodic pattern that maintains continuous monitoring coverage while preventing simultaneous execution of all tasks. This reduces peak load while preserving real-time monitoring capability.
Solution Approach 2:
The monitoring workload is segmented into multiple independent query tasks, each handling a specific portion of the monitoring responsibility. This segmentation allows the system to maintain high monitoring accuracy through frequent execution of individual tasks while preventing system destabilization by distributing the total load across multiple tasks with different execution schedules.
3Productivity
If more system resources are allocated to monitoring operations, then monitoring capability is enhanced, but the limited resource pool causes instability when resources are reserved for other storage services
Solution Approach 1:
The patent segments the monitoring workload into multiple smaller query tasks, each consuming fewer resources individually. This allows the system to enhance overall monitoring capability through parallel task execution while keeping resource consumption per task within acceptable limits, avoiding the need to allocate excessive resources to any single monitoring operation.
Solution Approach 2:
The patent employs partial action by having multiple query tasks each process a subset of the total data rather than one task processing everything. This distributed approach enhances monitoring coverage and capability while ensuring that no single task exceeds resource allocation limits, maintaining system stability despite enhanced monitoring requirements.
Data Source
AI summary
The technology described herein is directed towards executing continuous query tasks for monitoring a data storage system to obtain real-time metrics and store query results, in which the continuous query tasks are resource-demanding yet need to execute on limited system resources. A continuous query tasks scheduler levels the load produced by scheduling continuous query tasks with starting time offsets based on the period of continuous query tasks and load descriptions for the tasks. Further, a continuous query task that produces too high of a workload is divided into continuous query subtasks, which are scheduled at subtask starting time offsets.