Static Runtime Scheduling for Unplannable ITL Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing E-T-L processes struggle with unplannable workloads, particularly in I-T-L scenarios where source data is unknown and can be pushed at any time, frequency, or volume, leading to inefficiencies due to reserved processing capacity sitting idle or requiring excessive capacity.
Innovation Solution
A system and method that utilize a static runtime to efficiently schedule unplannable workloads by employing an ingestion service that sorts and batches incoming data, a scheduler that generates work orders based on batch thresholds, and runtime instances that execute these work orders dynamically, optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reserved processing capacity is maintained to handle unplannable I-T-L workloads, then system reliability is improved, but resource utilization deteriorates causing idle capacity
Solution Approach 1:
The system dynamically provisions processing capacity based on actual workload arrival. Instead of maintaining static reserved capacity, the system activates processing resources only when data arrives in the buffer, allowing capacity to adapt dynamically between idle and active states, thus resolving the contradiction between reliability and resource utilization
Solution Approach 2:
A buffer is introduced as an intermediary component between data sources and processing systems. The buffer absorbs incoming data and allows processing to occur at optimized intervals, decoupling the reliability requirement (data preservation) from the resource utilization constraint (processing activation), enabling the system to maintain reliability without permanent reserved capacity
2Adaptability or versatility
If large reserved processing capacity is allocated to handle multiple data types, then adaptability is improved, but cost increases due to excessive capacity requirements
Solution Approach 1:
The system segments processing capacity into specialized processing units, each optimized for specific data types or transformations. When a particular data type arrives, only the relevant processing unit is activated, providing adaptability for multiple data types while avoiding the need to maintain large reserved capacity for all possible types simultaneously
Solution Approach 2:
Processing units are designed with multi-functionality to handle various data types through configurable transformation rules. A single processing unit can be dynamically configured to process different data types, providing high adaptability without requiring separate dedicated capacity for each data type, thus reducing total capacity requirements
3Loss of time
If processing runs continuously on stand-by, then response time is improved, but operational cost increases due to idle capacity
Solution Approach 1:
Instead of continuous stand-by processing, the system employs periodic batch processing triggered by data accumulation in the buffer. Processing occurs in periodic batches when thresholds are met, eliminating the need for continuous idle operation while maintaining responsive processing capability, thus reducing operational costs without sacrificing response time
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for scheduling an unplannable workload via a static runtime. An ingestion service operating on a computing device establishes an inbound channel based on a setup order and associate the inbound channel to an Ingestion-Transformation-Load (ITL) task. The ingestion service stores incoming data received via the inbound channel in a staging area and organizes the incoming data into a plurality of batches. The ingestion service monitors the staging area to determine a number of unprocessed batches. Furthermore, in response to determining that the number of unprocessed batches meets or exceeds a first predetermined threshold, the ingestion service triggers a scheduler to generate a work order to be executed on runtime instance for each of the plurality of batches in the staging area.