Dynamic ETL Clustering for Full Resource Use and Failover

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing ETL systems face limitations in using 100% of available resources and ensuring high availability and scalability, particularly in multiplexed configurations, with existing solutions like Active-Standby duplexing and ETL JOB distributed management failing to address these needs.

Innovation Solution

A dynamic clustering-based ETL JOB distributed processing system that includes a cluster leader node distributing ETL operations to multiple normal nodes, processing the jobs, and loading them into a target DB, with automatic re-selection of the leader node in case of failure, ensuring 100% resource utilization and high availability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If Active-Standby duplexing is used for high availability, then system reliability is improved, but resource utilization deteriorates to only 50%

Engineering Contradiction:

Improvehigh availabilityVSAvoidresource utilization

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system segments the cluster into a leader node and multiple normal nodes, where the leader node manages ETL job distribution and normal nodes execute jobs. This segmentation allows multiple nodes to actively participate in processing, enabling resource utilization to exceed 50% while maintaining high availability through the leader's coordination and automatic failover capabilities.

Inventive Principle:

Principle #1Segmentation

2Productivity

If ETL JOB distributed management server is added for job distribution, then job distribution capability is improved, but system complexity increases and single point of failure risk increases

Engineering Contradiction:

Improvejob distribution capabilityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The leader node performs multiple functions including ETL job distribution, data extraction, and cluster management, eliminating the need for a separate dedicated job distribution server. This multi-functionality reduces system complexity while maintaining job distribution capability, and the distributed architecture across multiple nodes prevents single point of failure.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Loss of energy

If multiplexed ETL system configuration is implemented, then resource utilization is improved to 100%, but system complexity and architectural design difficulty increase

Engineering Contradiction:

Improveresource utilizationVSAvoidarchitectural design

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The system implements dynamic clustering where nodes can dynamically assume leader or normal roles based on operational needs and failure conditions. This dynamic allocation of roles simplifies the architectural design compared to static multiplexed configurations, while still achieving 100% resource utilization through flexible node assignment and automatic failover mechanisms.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12579161B2ETL job distributed processing system and method based on dynamic clustering

Publication Date: 2026.03.17 ARI SYST BLUE CO LTD

AI summary

According to an embodiment of the disclosure, an ETL JOB distributed processing method based on dynamic clustering includes: extracting, by a cluster leader node, data from a source DB; distributing, by the cluster leader node, an ETL (Extract, Transform, Load) JOB for performing an ETL operation to a plurality of cluster normal nodes based on the extracted data; processing, by the plurality of cluster normal nodes, the ETL JOB by calling the cluster leader node; and loading, by the cluster leader node and the plurality of cluster normal nodes, the processed ETL JOB into a target DB.