Partitioned Time-Series Scheduling for Ordered Cloud Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Time-series data captured from multiple sources over unreliable networks often arrives out of order, complicating customized scaling and processing in cloud-based databases, necessitating improved methods for ingestion and partitioning.
Innovation Solution
A system that allows clients to define custom partitioning and ordering definitions for time-series data, using a scheduler to manage execution pipelines, enqueue data based on partition keys, and block processing until ordering conditions are met, ensuring independent processing of different queues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If time-series data is processed in the order of arrival from unreliable networks, then processing speed is improved, but data ordering accuracy deteriorates
Solution Approach 1:
The system performs preliminary actions by assigning partition keys and enqueuing data samples into appropriate queues before processing. The scheduler pre-establishes the desired processing order through partitioning definitions and ordering definitions, so that when data arrives out of order from unreliable networks, it can be routed to the correct queue and processed in the proper sequence without requiring complex real-time reordering operations.
Solution Approach 2:
The system segments the data stream into multiple partition queues based on partition keys derived from ordering definitions. By dividing the single data stream into multiple ordered queues (e.g., queue 0, queue 1, queue 2), each handling a specific partition, the system maintains ordering within each queue while enabling parallel processing across queues, thus resolving the contradiction between processing speed and ordering accuracy.
2Manufacturing precision
If custom partitioning and ordering definitions are implemented, then data processing accuracy is improved, but system complexity increases
Solution Approach 1:
The system implements self-service by allowing clients to provide their own partitioning definitions and ordering definitions. The client application generates partition keys based on predefined ordering definitions (such as chronological order, sensor ID order, or custom business logic), and the scheduler automatically routes data samples to appropriate queues based on these keys. This eliminates the need for the server to implement complex custom partitioning logic, reducing system complexity while maintaining high processing accuracy.
Solution Approach 2:
The system reduces complexity by performing partitioning and ordering operations in advance. Clients pre-define partitioning schemes and ordering rules, and the scheduler pre-assigns partition keys to data samples before they are enqueued. This preliminary action ensures that data is correctly organized before processing begins, achieving high accuracy without requiring complex real-time decision-making in the processing path.
3Manufacturing precision
If data samples are blocked until ordering conditions are met, then data ordering accuracy is improved, but processing time increases
Solution Approach 1:
The system segments the data processing into multiple independent queues, each handling a specific partition. By dividing the data stream into parallel queues based on partition keys, the system can process multiple partitions simultaneously rather than sequentially blocking all data. This segmentation enables concurrent processing across queues while maintaining ordering within each queue, reducing overall processing time while preserving data ordering accuracy.
Solution Approach 2:
The system maintains continuity of useful action by allowing the scheduler to continue enqueuing and processing data samples across multiple queues without interruption. Even when certain samples need to wait for ordering conditions to be met in one queue, other queues can continue processing their respective partitions in parallel. This continuous processing across multiple streams minimizes idle time and reduces overall processing time while ensuring that each queue maintains its ordering integrity.
Data Source
AI summary
Scheduling of time-series data samples for processing according to client-specified criteria is performed. A client may submit a request to establish an execution pipeline to process time-series data, the request including a partitioning definition and an ordering definition. Responsive to this request, a scheduler may receive a time-series data sample and generate a partitioning key according to the partitioning definition to direct the sample to one of multiple partition queues. After enqueuing, the schedular may determine that the sample should be blocked from processing until satisfaction of an unblocking condition is achieved. This satisfaction may include processing of one or more other samples according to the ordering definition or a time period has elapsed. Blocking criteria may also include limitations in a number of simultaneously executing sample processes. Processing of samples of different queues may occur independently.