Attention mechanism-based load forecasting method and system
By employing an attention-based load forecasting method, which utilizes multi-head sparse self-attention and relative position encoding techniques to dynamically focus on time series, this method addresses the issues of forecast accuracy and adaptability in complex scenarios encountered by existing methods, achieving efficient and accurate power load forecasting.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA DATANG TECHNOLOGY INNOVATION CO LTD
- Filing Date
- 2024-12-02
- Publication Date
- 2026-05-21
AI Technical Summary
Existing power load forecasting methods struggle to meet accuracy and adaptability requirements in complex scenarios, especially with the integration of distributed energy resources and electric vehicles. Traditional methods fail to capture the long-term and short-term characteristics of the load, resulting in limited forecast accuracy.
A load forecasting method based on attention mechanism is adopted. The dependency relationship of long sequence input is learned through a multi-head probabilistic sparse self-attention model. Combined with relative position encoding technology, different parts of the time series are dynamically focused to extract important features and construct a multi-scale time series forecasting model for power load forecasting.
It improves the accuracy and adaptability of power load forecasting, enables real-time load forecasting in large-scale power grid environments, reduces computational complexity and cost, and adapts to various complex load patterns.
Smart Images
Figure CN2024136151_21052026_PF_FP_ABST
Abstract
Description
A load prediction method and system based on attention mechanism Technical Field
[0001] This invention belongs to the field of power technology, and in particular relates to a load forecasting method and system based on an attention mechanism. Background Technology
[0002] With the advancement of electricity market reforms and the construction of new power systems, the challenges facing the power grid are increasing. The integration of new loads such as distributed energy resources and electric vehicles has significantly increased the volatility and randomness of grid loads, placing higher demands on maintaining grid stability. Accurate load forecasting plays a crucial role in grid dispatching and management. It is not only the foundation for power system operation and management and generation planning, but also the key to ensuring that metering equipment operates at its optimal state and achieving accurate metering, control, and management.
[0003] Currently, power load forecasting methods are mainly divided into three categories: autoregressive models, statistical methods such as multiple linear regression and Kalman filtering, machine learning methods such as random forests and support vector machines, time series analysis methods, and simple neural network methods. While time series analysis methods have simple models and require less data, they are mainly suitable for stable time series and have limitations in handling complex nonlinear relationships and unstructured information. Machine learning methods have strong data modeling capabilities, but require extensive feature engineering and fail to fully capture the long-term and short-term characteristics of load, resulting in limited prediction accuracy. Simple neural network methods, such as backpropagation, deep learning, and long short-term memory networks, while possessing strong predictive capabilities, have high data requirements. In practical applications, insufficient data significantly reduces the learning and predictive capabilities of models, making it difficult to meet the needs of complex scenarios. Summary of the Invention
[0004] The purpose of this invention is to provide a load forecasting method and system based on an attention mechanism. Considering the diverse access resources in energy-consuming scenarios and the difficulty in accurately sensing the adjustable load range, existing power load forecasting methods are unable to meet the needs of current complex scenarios. By dynamically focusing on different parts of the time series through an attention mechanism, long-term dependencies in the time series can be captured, thereby more effectively extracting important features, capturing complex patterns in the data, improving the accuracy of power load forecasting, and providing decision support for clarifying the adjustable load range and optimizing energy allocation.
[0005] This invention provides a load prediction method based on an attention mechanism, comprising the following steps:
[0006] Step 1: Perform in-depth analysis on electricity consumption data, calculate the importance of each feature, select key features based on importance, learn the dependency relationship of long sequence input based on multi-head probabilistic sparse self-attention model, and use relative position encoding method to enhance the intrinsic connection between data to obtain data feature set;
[0007] Step 2: Based on the data feature set obtained in Step 1, the multi-scale time series prediction model is trained using the input feature set, and the load time series and peak-valley prediction are predicted. The multi-scale time series prediction model framework contains N layers, each layer contains multiple branches, which are used to learn the time series performance at different scales simultaneously. Each branch contains a labeler to convert the input representation into several segments, and the segments are processed into labels through self-attention mechanism and relative position encoding. The output representations of each branch in each layer of the framework are finally aggregated to form a single embedding and passed to the next layer.
[0008] Further, step 1 includes:
[0009] Step 1.1: Collect multi-timescale load data and multi-dimensional load data on the user side; the multi-timescale load data includes load data in units of hours, days, weeks, months, and years; the collected multi-dimensional load data on the user side includes historical load data, electricity, heat, cooling, and gas data, user behavior data, and electricity price data.
[0010] Step 1.2, Mining of Multi-Component Load Coupling Features:
[0011] Pearson correlation coefficient was used to analyze the intrinsic coupling relationship of multivariate loads and the correlation between multivariate loads and meteorological factors;
[0012] Step 1.3: Calculate feature importance and select key features:
[0013] For the raw load data, the original data matrix is expanded by taking the load ratio, load powers, and load logarithms, thus constructing the feature engineering of the raw data:
[0014] Step 1.4: Feature extraction is performed using a multi-head probabilistic sparse self-attention mechanism to capture long and short-term features.
[0015] Furthermore, the formula for calculating the Pearson correlation coefficient mentioned in step 1.2 is as follows:
[0016] In the formula: Let Y1 and Y2 be the Pearson correlation coefficients; cov(Y1,Y2) be the covariances of the eigenvectors Y1 and Y2. , Y1 and Y2 are the standard deviations, respectively; the correlation coefficient ranges from [-1, 1], the larger the absolute value of the correlation coefficient, the stronger the linear correlation; a correlation coefficient of 0.8-1.0 indicates a very strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.
[0017] Further, step 1.3 includes:
[0018] Based on the cold, heat, and electrical data, the expanded matrix of the original data is calculated as follows: Z = [Z1 Z2 Z3]
[0019] In the formula: Z1, Z2, and Z3 represent matrices after expanding the original data of cooling, heating, and power loads according to ratios, logarithms, and higher powers, respectively; each row in the matrix represents the data of a sample, where i = 1, 2, 3, ..., n in the i-th row; z EC,i z EH,i z CH,i Let z represent the electro-cooling ratio, electro-heating ratio, and cooling-heating ratio of the i-th sample, respectively; logE,i z logC,i z logH,i Let represent the logarithms of the electrical, cooling, and heating loads of the i-th sample, respectively. Let k represent the power of the electrical, cooling, and heating loads of the i-th sample, respectively; L represents the highest power series.
[0020] For matrices Z1 and Z2, Pearson correlation coefficients are directly used to analyze the correlation between the original electrical, cooling, and heating load data and the load ratios and logarithmic extended matrices. For matrix Z3, coupling matrices within and between loads are calculated respectively. For electrical loads:
[0021] In the formula: These are the coupling matrices between electrical loads and the coupling matrices between electrical loads and cooling / heating components, respectively; The element is in the i-th row and j-th column, where i,j = 1, 2, 3, ..., L; The matrix represents the correlation coefficient between the column vectors of the i-th and j-th powers of the electrical load; middle The correlation coefficient between the column vectors representing the i-th power of the electrical load and the j-th power of the cooling load, i,j=1,2,3,…,L; The correlation coefficient between the column vectors representing the i-th power of electrical load and the j-th power of thermal load, i,j=1,2,3,…,L;
[0022] The coupling characteristics of matrix Z3 are calculated using the following formula:
[0023] In the formula: Z b This represents the original data matrix of electrical load, L = 1; Z c Let Z represent the electrical load extended data matrix, L≠1; k=[(1 / 1!),(1 / 2!),…(1 / L!)] is the coefficient matrix required to establish a Taylor-like expansion; ⊙ represents the Hadamard product; F(E) is the coupling characteristic of matrix Z3; The coupling matrices between the electrical loads and between the electrical loads and the cooling and heating elements are given; where L is 3; Z... i The extended data matrix representing the electrical load encompasses data extended to different powers, used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k;
[0024] The obtained coupled feature variable matrix is:
[0025] In the formula: F(C) and F(H) are the coupling characteristics of the cold and heat loads obtained after calculation, respectively.
[0026] Further, step 1.4 includes:
[0027] The formula for calculating the self-attention model is:
[0028] In the formula: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q W k W v These represent the trainable linear transformation matrices corresponding to vectors Q, K, and V, respectively; d is the dimension of the input vector; and r is a scaling factor used to adjust the scale of the dot product between the key vector K and the query vector Q in the self-attention mechanism.
[0029] The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The formula for calculating the metric value is as follows:
[0030] In the formula: M(q) i ,K) is a measure of sparsity: q i k is the i-th element in Q; j L is the j-th element in K; k Let K be the length of K;
[0031] The formula for the probabilistic sparse self-attention mechanism is obtained based on the sparsity measurement method, namely:
[0032] In the formula: The probability sparse self-attention score matrix; For the first u largest M(q) i The set of query elements corresponding to K), where u = clnL Q c is the sampling factor, and in practical applications L Q =L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from O(L) to O(L). 2 The value is reduced to O(Ll nL);
[0033] Multi-head self-attention mechanisms divide the model's Q, K, and V into multiple branches, forming multiple subspaces. When combined with a sparse self-attention model, each subspace generates different sparse query-key pairs, allowing the model to focus on information from subspaces at different locations, thus avoiding information loss. The calculation formula is: A MH (Q,K,V)=Concat(h1,h2,…,h n W α
[0034] In the formula: A MH (Q,K,V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α h is a trainable linear transformation matrix. i This represents the attention score matrix for each head after computation using the sparse self-attention mechanism; Let be the self-attention score matrix;
[0035] In multi-head attention mechanisms, the linear transformation matrix of each head is a trainable parameter, and the model adaptively learns the sequence context that each head focuses on.
[0036] Further, step 2 includes:
[0037] Step 2.1, Branch-Specific Tokenization:
[0038] Within each layer of the model, each branch processes the input payload data using a fragment-based labeling method, transforming the input dataset... Divided into different branch fragment sets according to different granularities b represents the branch number; each branch contains segments of the same length, and each segment consists of multiple consecutive timestamps, representing a time step window; the size of the segment controls the granularity of the time features that the branch focuses on, with shorter segments used to capture high-frequency patterns, while longer segments are used to identify lower-frequency seasonal trends.
[0039] Step 2.2, Self-attention mechanism and relative position encoding:
[0040] In each branch, the self-attention mechanism from step 1.4 is applied to the tokenized fragments to identify dependencies between different time intervals; in each branch cut within each layer, the Transformer's self-attention mechanism is applied to the fragmented data. Obtain the output matrix Considering the periodicity of the load data and the distance between time steps, relative position encoding is applied in the model to enable the model to understand and predict recurring trends.
[0041] Step 2.3, Characterization Aggregation and Result Prediction:
[0042] In each layer, representations from all branches are combined into a single, unified result to be passed to the next layer; the output matrix of each branch... Flattened into a row vector b = 1, 2, ..., B; B is the total number of branches; the row vectors of all branches are then concatenated into a single total row vector. Finally, it is input into a linear layer to obtain the final result of that layer. Through this process, the last layer of the model outputs the predicted load value for future time steps.
[0043] Furthermore, step 2.1 also includes:
[0044] For each branch, the time series data is partially overlapped during the segmentation process to ensure that every part of the load data is taken into account during the forecasting process.
[0045] The present invention also provides a load forecasting system based on an attention mechanism, including a load forecasting module, wherein the load control module executes the load forecasting method based on an attention mechanism.
[0046] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor, implement the attention-based load prediction method.
[0047] The present invention also provides an electronic device, comprising:
[0048] The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the attention-based load prediction method.
[0049] By employing the above scheme, the load prediction method and system based on the attention mechanism achieve the following technical effects:
[0050] 1) This invention employs a multi-resolution attention mechanism, processing load data at different time scales through multiple branches. This allows for the accurate capture of short-term load fluctuations (such as daily peak-valley load changes) and the identification of long-term trends (such as seasonal variations). This technological advantage ensures the accuracy and timeliness of load forecasting.
[0051] 2) Compared to traditional load forecasting methods, this invention significantly optimizes computational complexity through a multi-timescale Transformer, effectively reducing the computational cost of large-scale load data while maintaining high forecasting accuracy. This advantage makes the method suitable for real-time load forecasting in large-scale power grid environments.
[0052] 3) This invention employs relative position encoding technology, enabling the model to automatically perceive the distance between time steps, thereby better capturing the periodic characteristics in the load data. This technological advantage allows the model to adaptively learn in different load scenarios and adapt to various complex load patterns.
[0053] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description
[0054] Figure 1 is a flowchart of the load prediction method of the attention mechanism of the present invention;
[0055] Figure 2 is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation
[0056] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0057] Referring to Figure 1, this embodiment provides a load prediction method based on an attention mechanism, including the following steps:
[0058] Step S1: Perform in-depth analysis on electricity consumption data, calculate the importance of each feature, select key features based on importance, learn the dependency relationship of long sequence input based on multi-head probabilistic sparse self-attention model, and use relative position encoding method to enhance the intrinsic connection between data to obtain data feature set;
[0059] Step S2: Based on the data feature set obtained in step S1, the multi-scale time series prediction model is trained using the input feature set, and the load time series and peak-valley prediction are predicted. The multi-scale time series prediction model framework contains N layers, each layer contains multiple branches, which are used to learn the time series performance at different scales simultaneously. Each branch contains a labeler to convert the input representation into several segments, and the segments are processed into labels through self-attention mechanism and relative position encoding. The output representations of each branch in each layer of the framework are finally aggregated to form a single embedding and passed to the next layer.
[0060] The present invention will now be described in further detail.
[0061] This attention-based load forecasting method mainly includes two aspects: perception and extraction of key multivariate load features, and forecasting of load time series and peak-valley patterns.
[0062] a) Perception and extraction of multiple key features
[0063] This paper utilizes an improved Transformer-based feature extraction algorithm for load data selection and a user key feature description algorithm to perform in-depth analysis and feature extraction on user electricity consumption data. The algorithm calculates the importance of each feature and selects key features based on this importance. It also learns the dependencies of long-sequence inputs through a multi-head probabilistic sparse self-attention model and employs a relative position encoding method to enhance the intrinsic connections between data, thereby improving the model's predictive performance. This optimizes the input feature set of the load forecasting model and enhances its prediction accuracy and robustness.
[0064] The specific operation and steps of this method are as follows:
[0065] Step 1.1 Collect multi-timescale load data from the user side, including load data in hours, days, weeks, months, and years; collect multi-dimensional load data from the user side, including historical load data, electricity, heat, cooling, gas data, user behavior data, and electricity price data, etc.
[0066] Step 1.2 Mining the Coupling Characteristics of Multiple Loads. The Pearson correlation coefficient (PCC) was used to analyze the intrinsic coupling relationships of multiple loads and their correlation with meteorological factors. The formula for calculating the Pearson correlation coefficient is:
[0067] In the formula: Let Y1 and Y2 be the Pearson correlation coefficients; cov(Y1,Y2) be the covariances of the eigenvectors Y1 and Y2. Let Y1 and Y2 be the standard deviations, respectively. The correlation coefficient ranges from -1 to 1. The larger the absolute value of the correlation coefficient, the stronger the linear correlation. Generally, a correlation coefficient of 0.8-1.0 indicates a very strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.
[0068] Step 1.3 Calculate feature importance and select key features. In Step 1.2, the Pearson coefficient can only describe the linear correlation between loads. To further explore the nonlinear coupling relationship between cooling, heating, and electrical loads, the original load data matrix is expanded by taking load ratios, load powers, and load logarithms to construct feature engineering for the original data. Taking cooling, heating, and electrical data as an example, the expanded matrix of the original data is calculated as follows: Z = [Z1 Z2 Z3]
[0069] In the formula: Z1, Z2, and Z3 represent matrices obtained by expanding the original data of cooling, heating, and electrical loads according to ratios, logarithms, and higher powers, respectively; z EC,i z EH,i z CH,i Let z represent the electro-cooling ratio, electro-heating ratio, and cooling-heating ratio of the i-th sample, respectively; logE,i z logC,i z logH,i Let represent the logarithms of the electrical, cooling, and heating loads of the i-th sample, respectively. Let k represent the power of the electrical, cooling, and heating loads of the i-th sample, respectively; L represents the highest power series.
[0070] For matrices Z1 and Z2, the Pearson correlation coefficient is directly used to analyze the correlation between the original electric, cooling, and heating load data and the load ratio and logarithmic extended matrix. For matrix Z3, the coupling matrices within and between loads are calculated respectively (taking electric load as an example).
[0071] In the formula: These are the coupling matrices between electrical loads and the coupling matrices between electrical loads and cooling / heating components, respectively; The element in the i-th row and j-th column (i,j = 1,2,3,…,L) The matrix represents the correlation coefficient between the column vectors of the i-th and j-th powers of the electrical load; middle (i,j=1,2,3,…,L) represents the correlation coefficient between the column vectors of the i-th power of the electrical load and the j-th power of the cooling load; (i,j=1,2,3,…,L) represents the correlation coefficient between the column vectors of the i-th power of electrical load and the j-th power of thermal load;
[0072] Next, the coupling characteristics of matrix Z3 are calculated using the following formula (taking electrical load as an example):
[0073] In the formula: Z b Represents the original data matrix of electrical load (L=1; Z) c Let L represent the electrical load extended data matrix (L≠1); k = [(1 / 1!), (1 / 2!), ..., (1 / L!)] is the coefficient matrix required to establish a Taylor-like expansion; ⊙ represents the Hadamard product. To avoid excessive computation, L is set to 3. F(E) represents the coupling characteristics of matrix Z3; Z represents the coupling matrix between the aforementioned electrical loads and the coupling matrix between the electrical loads and the cooling and heating elements; i The extended data matrix representing the electrical load covers data after different powers of expansion and is used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k.
[0074] Finally, the obtained coupled feature variable matrix is:
[0075] In the formula, F(C) and F(H) represent the coupling characteristics of the cold and hot loads obtained after calculation, respectively. The coupling feature mining algorithm used here can uncover the nonlinear coupling relationship between loads. In subsequent modeling, it can be used as an input variable for the prediction model to help improve the feature extraction effect of the model.
[0076] Step 1.4 utilizes a multi-head probabilistic sparse self-attention mechanism for feature extraction, capturing both short-term and long-term features. The self-attention model is currently the most widely used attention model, and its standard calculation formula is:
[0077] In the formula: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q W k W V d represents the trainable linear transformation matrices corresponding to vectors Q, K, and V, respectively; d is the dimension of the input vector; and r is a scaling factor used to adjust the scale of the dot product between the key vector K and the query vector Q in the self-attention mechanism.
[0078] The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The formula for calculating the metric value is as follows:
[0079] In the formula: M(q)i ,K) is a measure of sparsity: q i k is the i-th element in Q; j L is the j-th element in K; k Let K be the length of K;
[0080] The formula for the probabilistic sparse self-attention mechanism is obtained based on the sparsity measurement method, namely:
[0081] In the formula: The probability sparse self-attention score matrix; For the first u largest M(q) i The set of query elements corresponding to K), where u = clnL Q c is the sampling factor, and in practical applications L Q =L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from O(L) to O(L). 2 ) is reduced to O(Ll nL).
[0082] Multi-head self-attention mechanisms divide the model's Q, K, and V into multiple branches, forming multiple subspaces. When combined with a sparse self-attention model, each subspace generates different sparse query-key pairs, allowing the model to focus on information from subspaces at different locations, effectively avoiding information loss. The calculation formula is: A MH (Q,K,V)=Concat(h1,h2,…,h n W α
[0083] A MH (Q,K,V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α h is a trainable linear transformation matrix. i This represents the attention score matrix for each head after computation using the sparse self-attention mechanism; Let be the self-attention score matrix;
[0084] b) Load timing and peak-valley prediction
[0085] The data feature set extracted in a) is used to train and infer the multi-scale time series prediction model. Model overview: The load time series and peak-valley multi-time scale prediction model framework consists of N layers, each containing multiple branches. This multi-branch structure allows the model to learn time series representations at different scales simultaneously. Each branch contains a tokenizer to convert the input representation into patches. These patches are then processed into tokens through self-attention and relative position encoding. The output representations of each branch in each layer are finally aggregated to form a single embedding and passed to the next layer. The following is a detailed description of the model framework flow.
[0086] Step 2.1 Branch-Specific Labeling. Within each layer of the model, each branch processes the input payload data using a fragment-based labeling method. The input dataset... Divided into different branch fragment sets according to different granularities 'b' represents the branch number. Each branch contains segments of equal length, each segment consisting of multiple consecutive timestamps, representing a time step window. The size of the segment controls the granularity of the time features the branch focuses on. Branches with shorter segments are used to capture high-frequency patterns (such as daily load fluctuations), while branches with longer segments help identify lower-frequency seasonal trends. For each branch, the time series data is partially overlapped during the segmentation process to ensure that every part of the load data is considered during forecasting.
[0087] Step 2.2 Self-Attention Mechanism and Relative Position Encoding. In each branch, the self-attention mechanism from Step 1.4 will be applied to the tokenized fragments to identify dependencies between different time intervals. In each branch of each layer, as described above, the Transformer's self-attention mechanism will be applied to these fragmented data. Thus, the output matrix is obtained. This includes common techniques used in the model such as multi-head self-attention, residual connections, normalization, and feedforward networks. This mechanism allows the model to focus on the key time steps that have the greatest impact on future load forecasting. The self-attention mechanism is designed to operate on segments, making it ideal for capturing temporal dependencies within and between segments. Additionally, relative position encoding is applied to the model to account for the periodicity of load data. This encoding takes into account the distance between time steps, enabling the model to better understand and predict recurring trends, such as daily peaks or seasonal variations in load demand.
[0088] Step 2.3 Representation Aggregation and Result Prediction. In each layer, representations from all branches are combined into a single, unified result to be passed to the next layer. The output matrix of each branch... (b = 1, 2, ..., B; B is the total number of branches) is flattened into a row vector. The row vectors of all branches are then concatenated into a single total row vector. Finally, this data is fed into a linear layer to obtain the final result for that layer. This fusion process integrates short-term and long-term load patterns, enabling the model to make accurate load forecasts while taking into account immediate fluctuations and broader trends. Within this progressive framework, the final layer of the model outputs the predicted load values for future time steps. By leveraging a multi-timescale focus mechanism, the model is able to predict short-term peaks and troughs as well as long-term seasonal trends in load demand.
[0089] The attention-based load forecasting method has the following technical advantages:
[0090] 1) Employing a multi-resolution attention mechanism, load data at different time scales is processed through multiple branches, accurately capturing both short-term load fluctuations (such as daily peak-valley load changes) and identifying long-term trends (such as seasonal variations). This technological advantage ensures the accuracy and timeliness of load forecasting.
[0091] 2) Compared to traditional load forecasting methods, this method significantly optimizes computational complexity through multi-timescale Transformers, effectively reducing the computational cost of large-scale load data while maintaining high forecasting accuracy. This advantage makes this method suitable for real-time load forecasting in large-scale power grid environments.
[0092] 3) By employing relative position encoding technology, the model can automatically perceive the distance between time steps, thereby better capturing the periodic characteristics in the load data. This technological advantage enables the model to learn adaptively in different load scenarios and adapt to various complex load patterns.
[0093] This embodiment also provides a load forecasting system based on an attention mechanism, including a load forecasting module, which executes the load forecasting method based on an attention mechanism.
[0094] This embodiment also provides a non-transitory computer-readable storage medium that stores computer instructions, which, when executed by a processor, implement the attention-based load prediction method.
[0095] Referring to Figure 2, this embodiment also provides an electronic device, including:
[0096] The memory 201 and the processor 202 are interconnected and communicate with each other. The memory 201 stores computer instructions, and the processor 202 executes the attention-based load prediction method by executing the computer instructions.
[0097] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A load prediction method based on an attention mechanism, characterized in that, Includes the following steps: Step 1: Perform in-depth analysis on electricity consumption data, calculate the importance of each feature, select key features based on importance, learn the dependency relationship of long sequence input based on multi-head probabilistic sparse self-attention model, and use relative position encoding method to enhance the intrinsic connection between data to obtain data feature set; Step 2: Based on the data feature set obtained in Step 1, the multi-scale time series prediction model is trained using the input feature set, and the load time series and peak-valley prediction are predicted. The multi-scale time series prediction model framework contains N layers, each layer contains multiple branches, which are used to learn the time series performance at different scales simultaneously. Each branch contains a labeler to convert the input representation into several segments, and the segments are processed into labels through self-attention mechanism and relative position encoding. The output representations of each branch in each layer of the framework are finally aggregated to form a single embedding and passed to the next layer.
2. The load prediction method based on the attention mechanism according to claim 1, characterized in that, Step 1 includes: Step 1.1: Collect multi-timescale load data and multi-dimensional load data on the user side; the multi-timescale load data includes load data in units of hours, days, weeks, months, and years; the collected multi-dimensional load data on the user side includes historical load data, electricity, heat, cooling, and gas data, user behavior data, and electricity price data. Step 1.2, Mining of Multi-Component Load Coupling Features: Pearson correlation coefficient was used to analyze the intrinsic coupling relationship of multivariate loads and the correlation between multivariate loads and meteorological factors; Step 1.3: Calculate feature importance and select key features: For the raw load data, the original data matrix is expanded by taking the load ratio, load powers, and load logarithms, thus constructing the feature engineering of the raw data: Step 1.4: Feature extraction is performed using a multi-head probabilistic sparse self-attention mechanism to capture long and short-term features.
3. The load prediction method based on the attention mechanism according to claim 2, characterized in that, The formula for calculating the Pearson correlation coefficient mentioned in step 1.2 is as follows: In the formula: Let Y1 and Y2 be the Pearson correlation coefficients; cov(Y1,Y2) be the covariances of the eigenvectors Y1 and Y2. , Y1 and Y2 are the standard deviations, respectively; the correlation coefficient ranges from [-1, 1], the larger the absolute value of the correlation coefficient, the stronger the linear correlation; a correlation coefficient of 0.8-1.0 indicates a very strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.
4. The load prediction method based on the attention mechanism according to claim 3, characterized in that, Step 1.3 includes: Based on cold, heat, and electrical data, the expanded matrix of the original data is calculated as follows: Z=[Z1 Z2 Z3] In the formula: Z1, Z2, and Z3 represent matrices after expanding the original data of cooling, heating, and power loads according to ratios, logarithms, and higher powers, respectively; each row in the matrix represents the data of a sample, where i = 1, 2, 3, ..., n in the i-th row; z EC,i z EH,i z CH,i Let z represent the electro-cooling ratio, electro-heating ratio, and cooling-heating ratio of the i-th sample, respectively; logE,i z logC,i z logH,i Let represent the logarithms of the electrical, cooling, and heating loads of the i-th sample, respectively. Let k represent the power of the electrical, cooling, and heating loads of the i-th sample, respectively; L represents the highest power series. For matrices Z1 and Z2, Pearson correlation coefficients are directly used to analyze the correlation between the original electrical, cooling, and heating load data and the load ratios and logarithmic extended matrices. For matrix Z3, coupling matrices within and between loads are calculated respectively. For electrical loads: In the formula: These are the coupling matrices between electrical loads and the coupling matrices between electrical loads and cooling / heating components, respectively; The element is in the i-th row and j-th column, where i,j = 1, 2, 3, ..., L; The matrix represents the correlation coefficient between the column vectors of the i-th and j-th powers of the electrical load; middle The correlation coefficient between the column vectors representing the i-th power of the electrical load and the j-th power of the cooling load, i,j=1,2,3,…,L; The correlation coefficient between the column vectors representing the i-th power of electrical load and the j-th power of thermal load, i,j=1,2,3,…,L; The coupling characteristics of matrix Z3 are calculated using the following formula: In the formula: Z b This represents the original data matrix of electrical load, L = 1; Z c Let L represent the electrical load extended data matrix, L≠1; k=[(1 / 1!),(1 / 2!),…(1 / L!)] is the coefficient matrix required to establish a Taylor-like expansion; ⊙ represents the Hadamard product; F(E) is the coupling characteristic of matrix Z3; The coupling matrices between the electrical loads and between the electrical loads and the cooling and heating elements are given; where L is 3; Z... i The extended data matrix representing the electrical load encompasses data extended to different powers, used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k; The obtained coupled feature variable matrix is: In the formula: F(C) and F(H) are the coupling characteristics of the cold and heat loads obtained after calculation, respectively.
5. The load prediction method based on the attention mechanism according to claim 4, characterized in that, Step 1.4 includes: The formula for calculating the self-attention model is: In the formula: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q W k W v These represent the trainable linear transformation matrices corresponding to vectors Q, K, and V, respectively; d is the dimension of the input vector; and r is a scaling factor used to adjust the scale of the dot product between the key vector K and the query vector Q in the self-attention mechanism. The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The formula for calculating the metric value is as follows: In the formula: M(q) i ,K) is a measure of sparsity: q i k is the i-th element in Q; j L is the j-th element in K; k Let K be the length of K; The formula for the probabilistic sparse self-attention mechanism is obtained based on the sparsity measurement method, namely: In the formula: The probability sparse self-attention score matrix; For the first u largest M(q) i The set of query elements corresponding to K), where u = clnL Q c is the sampling factor, and in practical applications L Q =L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from O(L) to O(L). 2 The value is reduced to O(Ll nL); Multi-head self-attention mechanisms divide the model's Q, K, and V into multiple branches, forming multiple subspaces. When combined with a sparse self-attention model, each subspace generates different sparse query-key pairs, allowing the model to focus on information from subspaces at different locations, thus avoiding information loss. The calculation formula is as follows: A MH (Q,K,V)=Concat(h1,h2,…,h n )W α In the formula: A MH (Q,K,V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α h is a trainable linear transformation matrix. i This represents the attention score matrix for each head after computation using the sparse self-attention mechanism; Let be the self-attention score matrix; In multi-head attention mechanisms, the linear transformation matrix of each head is a trainable parameter, and the model adaptively learns the sequence context that each head focuses on.
6. The load prediction method based on the attention mechanism according to claim 5, characterized in that, Step 2 includes: Step 2.1, Branch-Specific Tokenization: Within each layer of the model, each branch processes the input payload data using a fragment-based labeling method, transforming the input dataset... Divided into different branch fragment sets according to different granularities b represents the branch number; each branch contains segments of the same length, and each segment consists of multiple consecutive timestamps, representing a time step window; the size of the segment controls the granularity of the time features that the branch focuses on, with shorter segments used to capture high-frequency patterns, while longer segments are used to identify lower-frequency seasonal trends. Step 2.2, Self-attention mechanism and relative position encoding: In each branch, the self-attention mechanism from step 1.4 is applied to the tokenized fragments to identify dependencies between different time intervals; in each branch cut within each layer, the Transformer's self-attention mechanism is applied to the fragmented data. Obtain the output matrix Considering the periodicity of the load data and the distance between time steps, relative position encoding is applied in the model to enable the model to understand and predict recurring trends. Step 2.3, Characterization Aggregation and Result Prediction: In each layer, representations from all branches are combined into a single, unified result to be passed to the next layer; the output matrix of each branch... Flattened into a row vector b = 1, 2, ..., B; B is the total number of branches; the row vectors of all branches are then concatenated into a single total row vector. Finally, it is input into a linear layer to obtain the final result of that layer. Through this process, the last layer of the model outputs the predicted load value for future time steps.
7. The load prediction method based on the attention mechanism according to claim 6, characterized in that, Step 2.1 further includes: For each branch, the time series data is partially overlapped during the segmentation process to ensure that every part of the load data is taken into account during the forecasting process.
8. A load prediction system based on an attention mechanism, characterized in that, The system includes a load forecasting module, which executes the attention-based load forecasting method according to any one of claims 1-7.
9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a processor, implement the attention-based load prediction method as described in any one of claims 1-7.
10. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform a load prediction method based on an attention mechanism as described in any one of claims 1-7.