Medical ETL Task Dispatching via Dynamic Executor Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current medical data ETL systems fail to effectively utilize heterogeneous cluster resources and differentiate resource demands across different stages of ETL tasks, leading to suboptimal dispatching performance and throughput in multi-center environments.
Innovation Solution
A two-level dispatching mechanism is implemented, where a dispatching machine collects resource index vectors and task load information to select the most suitable executor for task execution, utilizing a time prediction equation to prioritize tasks and manage active and expired queues, ensuring efficient resource allocation and re-dispatching based on stage characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If ETL tasks are sent to a cluster environment for dispatching and execution, then the computing power limitation of individual medical institutions is overcome, but the utilization of heterogeneous cluster resources is insufficient
Solution Approach 1:
The patent applies local quality by matching specific task stages with executors that have appropriate resource characteristics. Different executors in the heterogeneous cluster are assigned to different task stages based on their CPU, GPU, and memory capabilities, ensuring that each executor operates in its optimal performance zone and maximizing overall cluster resource utilization.
Solution Approach 2:
The patent implements dynamic dispatching where the system continuously monitors executor status and task requirements, then dynamically assigns tasks to appropriate executors. The dispatching mechanism adapts to changing cluster conditions and task characteristics in real-time, optimizing resource allocation throughout the ETL process rather than using static assignment.
2Reliability
If multiple sub-stages are executed as a whole task, then the sequential dependence relationship is maintained, but the adaptability to different machine resources is reduced
Solution Approach 1:
The patent segments the ETL task into multiple independent sub-stages (data extraction, data cleaning, data transformation, data loading) that can be executed separately on different executors. Each sub-stage is assigned to an executor best suited for its specific resource requirements, while the system maintains the sequential dependence relationships through proper task scheduling and state management.
Solution Approach 2:
Different sub-stages are assigned to executors with matching resource characteristics. For example, CPU-intensive extraction stages are assigned to executors with strong processing power, while memory-intensive transformation stages are assigned to executors with adequate memory resources, optimizing the adaptability of each stage to its execution environment.
3Power
If a single stage has large dependence on certain machine resources, then the task can be optimized for that resource, but the overall dispatching performance in heterogeneous clusters is reduced
Solution Approach 1:
The patent changes the dispatching parameters by considering multiple factors including executor resource characteristics, task stage requirements, and current cluster load conditions. The dispatching algorithm dynamically adjusts assignment decisions based on these parameters, balancing resource efficiency for individual stages with overall cluster dispatching performance through multi-criteria optimization.
Data Source
AI summary
The present disclosure discloses a medical ETL task dispatching method, system and apparatus based on multiple centers. The method includes following steps: step S1: testing and verifying ETL tasks; step S2: deploying the ETL tasks to a hospital center, and dispatching the ETL tasks to a plurality of executors for execution; step S3: screening an executor set meeting resource demands of ETL tasks to be dispatched; step S4: calculating a current task load of each executor in the executor set; step S5: selecting the executor with a minimum current task load to execute the ETL tasks; and step S6: selecting, by the dispatching machine, the ETL tasks from executor active queues according to a priority for execution. The present disclosure selects the most suitable executor by analyzing a serving index as a task to be dispatched on a current dispatching machine.