This application relates to the field of ETL resource optimization technology, specifically to a method,
system, equipment, and medium for
dynamic resource optimization of ETL pipelines. The method includes: collecting time-
series data of the ETL pipeline
operating environment, dividing it into
time windows, and then standardizing it; generating a
linear prediction sequence of
system load indicators using a linear load prediction model; generating a coupled prediction sequence of
system load indicators using a pre-trained multi-dimensional time-series load prediction model; integrating the two prediction sequences and then performing de-
standardization to obtain a comprehensive prediction sequence of the original system load indicators; identifying peak periods, off-peak periods, and periods of abnormal
resource scarcity based on the comprehensive prediction sequence of the original system load indicators; and dynamically adjusting and executing the scheduling strategy of ETL tasks based on the identification results. This application achieves automated optimization of ETL pipeline resources and improved system efficiency by collecting multi-source time-
series data, combining two models to generate a comprehensive prediction sequence, and dynamically adjusting the scheduling strategy.