Method for multi-modal data processing in electric power dispatching, and system, device and storage medium

Abstract

A method for multi-modal data processing in electric power dispatching, and a system, a device and a storage medium. The method comprises the steps of data collection and preprocessing, multi-modal data annotation and sample library construction, feature extraction and fusion, entity recognition and extraction, knowledge graph construction, etc., thereby effectively processing text, image, audio and video data in an electric power dispatching system. BERT and ResNet models are used to extract features, a cross-modal attention mechanism is used to fuse the features, a graph neural network and a conditional random field model are used to perform entity recognition and optimization, and finally, an optimized knowledge graph is constructed.

Description

Multimodal data processing methods, systems, equipment and storage media for power dispatching Technical Field

[0001] This disclosure belongs to the field of power dispatching technology, and specifically relates to a power dispatching multimodal data processing method, system, equipment and storage medium. Background Technology

[0002] As the central nervous system of the power grid, the power dispatching system is crucial for ensuring the safe and stable operation of the power grid. In power dispatching, dispatchers need to process data from various sources in real time, including voice, text, video, and images, to understand the grid status, make dispatching decisions, and troubleshoot faults. With the continuous expansion of the power grid and the advancement of technology, the amount and complexity of data processed by the power dispatching system are constantly increasing, placing ever higher demands on dispatchers.

[0003] Therefore, constructing a multimodal knowledge graph for intelligent auxiliary decision-making in power dispatching systems to enable intelligent assistance to dispatchers is a current research hotspot and important trend.

[0004] Currently, the power dispatching system has the following problems:

[0005] 1) Lack of a unified processing method for multimodal data: Power dispatching systems contain multimodal data such as voice, text, images and video. Different modal data are processed in different ways, making it impossible to achieve unified processing and fusion of data, resulting in low data processing efficiency and affecting the work efficiency of dispatchers.

[0006] 2) Lack of cross-modal consistency in entity recognition: There are various entities in the power dispatching system, such as equipment and personnel. The representation of entities in different modal data is different. For example, entities in speech are speech segments, entities in text are text segments, and entities in images and videos are image or video segments. The lack of a unified representation of entities in different modal data leads to poor accuracy and consistency in entity recognition, which affects the accuracy of dispatching decisions.

[0007] 3) Lack of Knowledge Graph Construction for Entity Attributes and Relationships: Power dispatching systems contain entity attributes and relationships between entities, such as equipment model, service life, operating status, and connections between equipment. Currently, these entity attributes and relationships lack knowledge graph construction, making it impossible to achieve a comprehensive description of entities and a representation of the connections between them, thus affecting the intelligent auxiliary decision-making capabilities of the dispatching system.

[0008] 4) Lack of efficient processing and knowledge graph construction methods for multimodal data: Currently, the processing methods for multimodal data in power dispatching systems are not efficient enough, and there is a lack of methods for cross-modal fusion and knowledge graph construction of multimodal data, which affects the intelligence level of power dispatching systems.

[0009] Therefore, how to construct a multimodal knowledge graph for intelligent auxiliary decision-making in power dispatching systems, and how to achieve efficient processing of multimodal data and knowledge graph construction, are problems that need to be solved in this field.

[0010] Summary of the Invention

[0011] This disclosure provides a method, system, device, and storage medium for processing multimodal data in power dispatching, aiming to at least address the technical problems of how to construct a multimodal knowledge graph for intelligent auxiliary decision-making in power dispatching systems and achieve efficient processing of multimodal data and construction of the knowledge graph.

[0012] According to a first aspect of the present disclosure, a multimodal data processing method for power dispatching is provided, comprising: multimodal data acquisition, acquiring diverse heterogeneous data including text, images, audio, and video from a power dispatching system; data preprocessing, converting audio data using natural language processing methods, preprocessing images using OpenCV, and performing frame segmentation processing on video data; multimodal data annotation and sample library construction, processing the text data using the BIO annotation specification, performing entity detection and annotation of the image and video data using CNN, and constructing a unified knowledge sample library; feature extraction and fusion, extracting text features using a BERT model, extracting image features using an improved ResNet model, and fusing features from different modalities through a cross-modal attention mechanism; entity recognition and extraction, performing entity recognition using a graph neural network (GNN), and optimizing the entity labels using a conditional random field (CRF) model to improve the coherence and accuracy of recognition; and knowledge graph construction, establishing entity attribute sets and relationships between entities based on the entity recognition results, and optimizing the knowledge graph using graph embedding technology.

[0013] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0014] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0015] Optionally, the multimodal data processing method for power dispatch also includes: cross-modal feature consistency adjustment: before feature fusion, the feature vectors of different modalities are adjusted to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; multi-head attention mechanism optimization: using a multi-head attention mechanism to increase the processing depth of the model for different feature vectors and improve the accuracy of feature extraction and fusion.

[0016] Optionally, the power dispatch multimodal data processing method also includes: entity label sequence optimization: using a conditional random field model to optimize the entity label sequence output by the graph neural network to ensure the consistency and accuracy of entity recognition results; automatic entity attribute extraction: after entity recognition, automatically extract entity attributes from the multimodal data, including equipment type, operating status, etc., to enrich the entity information of the knowledge graph.

[0017] Optionally, the multimodal data processing method for power dispatch also includes: dynamic updating of entity relationships: dynamically updating entity relationships in the knowledge graph according to real-time changes in the power system to ensure the timeliness and accuracy of the knowledge graph; selection of graph embedding optimization algorithm: selecting a graph embedding optimization algorithm suitable for the power dispatch scenario, and optimizing the embedding representation of entities and relationships according to the algorithm characteristics and task requirements.

[0018] According to a second aspect of the present disclosure, a power dispatching multimodal data processing system is also provided, comprising: an acquisition module configured for multimodal data acquisition, acquiring diverse heterogeneous data including text, images, audio, and video from a power dispatching system; a processing module configured for data preprocessing, converting audio data using natural language processing methods, preprocessing images using OpenCV, and performing frame segmentation processing on video data; and a first construction module configured for multimodal data annotation and sample library construction, processing the text data using the BIO annotation specification, and performing entity detection and annotation of the image and video data using CNN. A unified knowledge sample base is established; a fusion module is set up for feature extraction and fusion, using the BERT model to extract text features and the improved ResNet model to extract image features, and fusing features from different modalities through a cross-modal attention mechanism; a recognition module is set up for entity recognition and extraction, using a graph neural network (GNN) for entity recognition and using a conditional random field (CRF) model to optimize the entity labels, thereby improving the coherence and accuracy of recognition; a second construction module is set up for knowledge graph construction, establishing entity attribute sets and relationships between entities based on the entity recognition results, and optimizing the knowledge graph using graph embedding technology.

[0019] According to a third aspect of the present disclosure, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the power dispatch multimodal data processing method described in any of the embodiments of the first aspect above.

[0020] According to a fourth aspect of the present disclosure, a non-volatile storage medium is also provided, wherein a computer program is stored in the non-volatile storage medium, and the computer program is configured to execute the power dispatch multimodal data processing method described in any embodiment of the first aspect when running on a computer or processor.

[0021] According to a fifth aspect of the present disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the power dispatch multimodal data processing method described in any of the embodiments of the first aspect.

[0022] In this embodiment, the power dispatch multimodal data processing method of this application effectively improves the processing efficiency and accuracy of diverse and heterogeneous data in the power dispatch system by comprehensively utilizing natural language processing, computer vision, and deep learning technologies. Specifically, by combining cross-modal attention mechanisms and graph neural networks, entity information can be identified and extracted more accurately, and the constructed knowledge graph is more comprehensive and precise, providing more intelligent and reliable data support for power dispatch. Furthermore, the application of data augmentation and unified annotation standards significantly improves the model's generalization ability and the consistency of entity recognition, enabling the system to make more accurate decisions when facing complex and ever-changing power dispatch environments. Overall, this solution can significantly improve the intelligence level of power dispatch, reduce human intervention, increase dispatch efficiency, reduce power system operation risks, and bring significant economic and social benefits to the power industry. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are intended to explain this disclosure, but do not constitute an undue limitation thereof. In the drawings:

[0024] Figure 1 is a flowchart of a power dispatch multimodal data processing method according to one embodiment of the present disclosure;

[0025] Figure 2 is a structural block diagram of a power dispatch multimodal data processing system according to one embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are intended to distinguish similar objects and are not necessarily intended to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of this disclosure, an embodiment of a power dispatch multimodal data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system containing at least one set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This method embodiment can also be executed in an electronic device including a memory and a processor, a similar control device, or in the cloud. Taking an electronic device as an example, the electronic device may include one or more processors and a memory configured to store data. Optionally, the electronic device may also include a communication device configured for communication functions and a display device. Those skilled in the art will understand that the above structural description is merely illustrative and does not limit the structure of the electronic device. For example, the electronic device may include more or fewer components than described above, or have a different configuration than described above.

[0030] A processor may include one or more processing units. For example, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microcontroller unit (MCU), a field-programmable gate array (FPGA), a neural network processing unit (NPU), a tensor processing unit (TPU), or an artificial intelligence (AI) processor. Different processing units may be independent components or integrated into one or more processors. In some instances, electronic devices may also include one or more processors.

[0031] The memory may be configured to store computer programs, such as the computer program corresponding to the power dispatch multimodal data processing method in this embodiment. The processor implements the aforementioned power dispatch multimodal data processing method by running the computer program stored in the memory. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0032] The communication device is configured to receive or transmit data via a network. Specific examples of the network mentioned above may include a wireless network provided by the mobile terminal's communication provider. In one example, the communication device includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the communication device may be a radio frequency (RF) module, configured to communicate with the Internet wirelessly. In some embodiments of this solution, the communication device is configured to connect to mobile devices such as mobile phones and tablets, enabling the mobile device to send commands to the electronic device.

[0033] The display device can be a touchscreen liquid crystal display (LCD) or a touch display (also referred to as a "touchscreen" or "touch display screen"). The LCD allows a user to interact with the user interface of the electronic device. In some embodiments, the electronic device has a graphical user interface (GUI), which allows the user to interact with the GUI by touching a touch-sensitive surface with fingers and / or gestures. Executable instructions configured to perform the aforementioned human-computer interaction functions are configured / stored in one or more processor-executable computer program products or readable storage media.

[0034] Figure 1 is a flowchart of a power dispatch multimodal data processing method according to one embodiment of the present disclosure. As shown in Figure 1, it includes the following steps:

[0035] Step S101: Multimodal data acquisition, obtaining diverse and heterogeneous data including text, images, audio and video from the power dispatching system.

[0036] Step S102: Data preprocessing, using natural language processing methods to convert audio data, using OpenCV to preprocess images, and performing frame segmentation on video data.

[0037] Step S103: Multimodal data annotation and sample library construction. The text data is processed using the BIO annotation specification, and entity detection and annotation of the image and video data are performed using CNN to construct a unified knowledge sample library.

[0038] Step S104, Feature Extraction and Fusion: The text features are extracted using the BERT model, the image features are extracted using the improved ResNet model, and the features from different modalities are fused through a cross-modal attention mechanism.

[0039] Step S105, Entity recognition and extraction: Graph Neural Network (GNN) is used for entity recognition, and Conditional Random Field (CRF) model is used to optimize the entity labels to improve the consistency and accuracy of recognition.

[0040] Step S106: Knowledge graph construction. Based on the entity recognition results, establish entity attribute sets and relationships between entities, and optimize the knowledge graph using graph embedding technology.

[0041] In this embodiment, the power dispatch multimodal data processing method of this application effectively improves the processing efficiency and accuracy of diverse and heterogeneous data in the power dispatch system by comprehensively utilizing natural language processing, computer vision, and deep learning technologies. Specifically, by combining cross-modal attention mechanisms and graph neural networks, entity information can be identified and extracted more accurately, and the constructed knowledge graph is more comprehensive and precise, providing more intelligent and reliable data support for power dispatch. Furthermore, the application of data augmentation and unified annotation standards significantly improves the model's generalization ability and the consistency of entity recognition, enabling the system to make more accurate decisions when facing complex and ever-changing power dispatch environments. Overall, this solution can significantly improve the intelligence level of power dispatch, reduce human intervention, increase dispatch efficiency, reduce power system operation risks, and bring significant economic and social benefits to the power industry.

[0042] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0043] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0044] Optionally, the multimodal data processing method for power dispatch also includes: cross-modal feature consistency adjustment: before feature fusion, the feature vectors of different modalities are adjusted to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; multi-head attention mechanism optimization: using a multi-head attention mechanism to increase the processing depth of the model for different feature vectors and improve the accuracy of feature extraction and fusion.

[0045] Optionally, the power dispatch multimodal data processing method also includes: entity label sequence optimization: using a conditional random field model to optimize the entity label sequence output by the graph neural network to ensure the consistency and accuracy of entity recognition results; automatic entity attribute extraction: after entity recognition, automatically extract entity attributes from the multimodal data, including equipment type, operating status, etc., to enrich the entity information of the knowledge graph.

[0046] Optionally, the multimodal data processing method for power dispatch also includes: dynamic updating of entity relationships: dynamically updating entity relationships in the knowledge graph according to real-time changes in the power system to ensure the timeliness and accuracy of the knowledge graph; selection of graph embedding optimization algorithm: selecting a graph embedding optimization algorithm suitable for the power dispatch scenario, and optimizing the embedding representation of entities and relationships according to the algorithm characteristics and task requirements.

[0047] Specifically, the steps of the above method include:

[0048] Step 1: Data Acquisition and Preprocessing

[0049] 1. Multi-source data acquisition

[0050] Collect diverse and heterogeneous data in the field of power dispatching, including text, images, audio, and video data.

[0051] 2. Data Preprocessing

[0052] Text data preprocessing: Extract plain text content, remove redundant information, and standardize the format.

[0053] Natural language processing methods are used to convert audio into text data.

[0054] Data deduplication, data filtering, and data augmentation can be performed using OpenCV for image processing.

[0055] Video data preprocessing: The video data is processed frame by frame, converted into multi-frame image data, and then processed according to the image preprocessing method.

[0056] Step 2: Multimodal data annotation and sample library construction

[0057] 1. Multimodal data annotation

[0058] Specifically, this includes text data annotation, image data annotation, and video data annotation.

[0059] (1) Text data annotation

[0060] B: Indicates the beginning of an entity (Begin).

[0061] I: Indicates the interior of an entity.

[0062] O: Represents a non-entity (Outside).

[0063] Let T = {t1, t2, ..., t} n} is a text sequence, t i This represents the i-th character.

[0064] The annotation result is L={l1,l2,...,l n}, where l i For the label of the i-th character, l i ∈{B-Entity,T-Entity,O}.

[0065] (2) Image data annotation

[0066] Convolutional Neural Networks (CNNs) are used for object detection and classification, extracting entities from images and labeling them.

[0067] Let I be the input image. The detection result is D = {(x i ,y i ,w i ,h i ,c i )}, where (x i ,y i ) represents the top-left corner coordinate of the detected i-th object, w i and h i These represent the width and height of the object, respectively. i Indicates the category of the object.

[0068] (3) Video data annotation

[0069] The video data is processed in frames at certain time intervals to convert it into multi-frame image data, and then labeled according to the image data annotation method.

[0070] Let V be the input video, and the frame rate be f.

[0071] After framing, the resulting frame sequence is F = {I1, I2, ..., I...} m}, where I j Let j be the image of the j-th frame.

[0072] 2. Construction of Knowledge Sample Base

[0073] The labeled multimodal data is stored in a unified knowledge sample base, which facilitates subsequent feature extraction and model training.

[0074] (1) Sample database data format

[0075] a, Text sample library

[0076] Store the annotated text sequence and its corresponding label sequence.

[0077] Let {(T1,L1),(T2,L2),...,(T k ,L k )} is a text sample library, T i For the i-th sample, L i It is its corresponding label sequence.

[0078] b, Image sample library

[0079] Store images and their object detection and classification information.

[0080] Let {(I1,D1),(I2,D2),...,(I n D n )} is an image sample library, I i For the i-th image, D i The results of the object detection.

[0081] c, Video sample library

[0082] Store the video frame images after framing and their annotation information.

[0083] Let {(V1,F1),(V2,F2),...,(V k ,F k )} represents a video sample library, V i For the i-th video segment, F i It is the image frame sequence after framing and its annotation information.

[0084] Step 3: Feature Extraction and Fusion

[0085] 1. Feature Extraction

[0086] (1) Text data feature extraction

[0087] Use the BERT model to extract contextual semantic features from text data.

[0088] Let T = {t1, t2, ..., t} n} is a text sequence, t i This represents the i-th character.

[0089] The BERT model outputs a feature vector H = {h1, h2, ..., h} for each word. n}, where h i Let be the feature vector of the i-th character.

[0090] (2) Feature extraction from image data

[0091] Image features were extracted using an improved ResNet model.

[0092] Let I be the input image.

[0093] The output of the ResNet model is the feature vector F = {f1, f2, ..., f3} of the image. k}, where f i Let i be the i-th image feature.

[0094] (3) Feature extraction of converted audio text data

[0095] You can refer to the text data feature extraction steps and use the BERT model to extract the contextual semantic features of the text data.

[0096] Let T' = {t1', t'2, ..., t'} m} represents the converted audio text sequence, t i ' represents the i-th character.

[0097] The BERT model outputs a feature vector H' = {h1', h'2, ..., h'} for each word. m}, where h i ' represents the feature vector of the i-th character.

[0098] 2. Cross-modal attention mechanism

[0099] (1) Vector encoding

[0100] Data from different modalities is converted into vector-encoded representations. The text feature vectors are H and H', and the image feature vector is F.

[0101] (2) Cross-modal attention mechanism

[0102] A cross-modal attention mechanism is used to fuse features from different modalities.

[0103] Let Q, K, and V represent the query vector, key vector, and value vector, respectively. For each feature vector, calculate the attention weights:

[0104] Where, d k It is the dimension of the key vector.

[0105] Calculate multi-head attention by incorporating the multi-head attention mechanism:

[0106] MultiHead(Q,K,V)=Concat(head1,head2,...,head h W O

[0107] Where head1 = Attention(QW i Q ,KW i K VW i V ), W O W is the projection matrix. i Q W represents the query projection matrix of the i-th head. i K W represents the key projection matrix of the i-th head. iV The projection matrix represents the value of the i-th head.

[0108] 3. Feature fusion:

[0109] The fused feature vectors are then subjected to unified feature encoding to generate the final fused feature representation.

[0110] Let the fused feature vector be H. F .

[0111] H F =MultiHead(H,H,H)+MultiHead(F,F,F)+MultiHead(H',H',H')

[0112] The final feature is represented as:

[0113] F final =LayerNorm(H F +FeedForward(H F ))

[0114] Here, LayerNorm represents layer normalization, and FeedForward represents a feedforward neural network.

[0115] Step 4: Entity Recognition and Extraction:

[0116] 1. Entity recognition

[0117] (1) Graph Neural Network (GNN)

[0118] Graph neural networks are used to further process multimodal features and capture the relationships between them.

[0119] H (0) =F final

[0120] Among them, H (0) H is the initial input feature. (l) Let be the feature representation of the i-th layer of the graph neural network, and σ be the activation function.

[0121] (2) Entity recognition output

[0122] The graph neural network outputs entity recognition result O.

[0123] O = softmax(H) L W O )

[0124] 2. Entity extraction

[0125] Using the output H of the graph neural network LThe entity recognition result is O. The entity labels are then optimized using a conditional random field model.

[0126] Where y is the feature sequence, h i For the i-th feature, Ψ(y) i ,h i ) is the characteristic function, φ(y) i ,y i+1 ) is the transfer function.

[0127] Sequence optimization is performed, and the optimal label sequence y is solved using the Viterbi algorithm. * .

[0128] Step 5: Knowledge Graph Construction:

[0129] 1. Establishment of entity relationships

[0130] Based on the identified and extracted entity e i Establish its attribute set A i .

[0131] Let E = {e1, e2, ..., e} n Let A be the set of identified entities. i For entity e i The set of attributes.

[0132] A i ={a i1 ,a i2 ,...,a im}

[0133] Among them, a ij Represents entity e i The j-th attribute.

[0134] Then, based on the power system topology and the physical connections between the devices, the relationships between entities are established. ij .

[0135] Let R = {r1, r2, ..., r} k Let} be the set of relationships between entities, and r ij Represents entity e i and entity e j The relationship between them.

[0136] r ij = <e i ,e j ,t ij >

[0137] Among them, t ijIndicates the type of relationship, such as "connect", "contain", "control", etc.

[0138] 2. Knowledge graph optimization

[0139] First, let's look at knowledge unit k. i Perform semantic alignment and mapping to ensure the accuracy of the knowledge graph.

[0140] W k ={w k1 ,w k2 ,...,w kn}

[0141] W k It is knowledge unit k i The set of word vectors. w ki Represents knowledge unit k i Word vectors, by mapping knowledge units to a vector space, can quantify and compare the semantic relationships between different knowledge units, ensuring the accuracy and consistency of the knowledge graph.

[0142] Secondly, it integrates multimodal regulation entity knowledge to form a multimodal knowledge graph.

[0143] L ij = <k i ,k j ,s ij >

[0144] L ij Represents knowledge unit k i and k j The connection relationship between them. ij The type and weight of the link indicate the strength and type of the association between knowledge units.

[0145] Then, graph embedding techniques are used to optimize the knowledge graph, improving its performance in complex query and reasoning tasks.

[0146] Entity embedding representation: E = {e1, e2, ..., e n}

[0147] E represents the set of embedded representations of all entities. By embedding entities into a low-dimensional space, computation and storage can be performed more efficiently, while preserving information about the relationships between entities.

[0148] Relational embedding representation: R = {r} 11 ,r 12 ,...,r nn}

[0149] R represents the set of embedding representations of all relations. ij It is entity e i and entity ej The low-dimensional vector representation of relationships between entities allows for more efficient computation of relationships between entities by embedding relationships into a low-dimensional space.

[0150] Graph embedding optimization objective:

[0151] This formula represents the optimization objective of graph embedding, which is to minimize the distance function between entities and relations, ensuring that the embedding representation accurately reflects the semantics of entities and relations.

[0152] This indicates that the summation is performed on all entity relation pairs.

[0153] ||e i +r ij -e j || 2 Represents entity e i and entity e j The distance between relationships is minimized to ensure that the embedded representations of entities and relationships accurately reflect the semantic information in the knowledge graph.

[0154] By embedding entities and relations in knowledge graphs and optimizing the distance between embedding vectors, the performance of knowledge graphs in complex query and reasoning tasks can be improved.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0156] This embodiment also provides a power dispatch multimodal data processing system, which is configured to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" refers to a combination of software and / or hardware capable of performing a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0157] Figure 2 is a structural block diagram of a power dispatch multimodal data processing system 200 according to one embodiment of the present disclosure. As shown in Figure 2, the power dispatch multimodal data processing system 200 is used as an example, including: an acquisition module 201, configured for multimodal data acquisition, acquiring diverse heterogeneous data including text, images, audio, and video from the power dispatch system; a processing module 202, configured for data preprocessing, converting audio data using natural language processing methods, preprocessing images using OpenCV, and performing frame segmentation processing on video data; and a first construction module 203, configured for multimodal data annotation and sample library construction, processing the text data using the BIO annotation specification, and using CNN to process the images. The system includes: entity detection and annotation of image and video data to build a unified knowledge sample base; a fusion module 204 for feature extraction and fusion, which uses the BERT model to extract text features and the improved ResNet model to extract image features, and fuses features from different modalities through a cross-modal attention mechanism; a recognition module 205 for entity recognition and extraction, which uses a graph neural network (GNN) for entity recognition and uses a conditional random field (CRF) model to optimize the entity labels, thereby improving the coherence and accuracy of recognition; and a second construction module 206 for knowledge graph construction, which establishes entity attribute sets and relationships between entities based on the entity recognition results, and optimizes the knowledge graph using graph embedding technology.

[0158] Embodiments of this disclosure also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the power dispatch multimodal data processing method described in any of the above embodiments.

[0159] Optionally, in this embodiment, the processor in the above-described electronic device may be configured to run a computer program to perform the following steps:

[0160] Step S101: Multimodal data acquisition, obtaining diverse and heterogeneous data including text, images, audio and video from the power dispatching system.

[0161] Step S102: Data preprocessing, using natural language processing methods to convert audio data, using OpenCV to preprocess images, and performing frame segmentation on video data.

[0162] Step S103: Multimodal data annotation and sample library construction. The text data is processed using the BIO annotation specification, and entity detection and annotation of the image and video data are performed using CNN to construct a unified knowledge sample library.

[0163] Step S104, Feature Extraction and Fusion: The text features are extracted using the BERT model, the image features are extracted using the improved ResNet model, and the features from different modalities are fused through a cross-modal attention mechanism.

[0164] Step S105, Entity recognition and extraction: Graph Neural Network (GNN) is used for entity recognition, and Conditional Random Field (CRF) model is used to optimize the entity labels to improve the consistency and accuracy of recognition.

[0165] Step S106: Knowledge graph construction. Based on the entity recognition results, establish entity attribute sets and relationships between entities, and optimize the knowledge graph using graph embedding technology.

[0166] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0167] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0168] Optionally, the multimodal data processing method for power dispatch also includes: cross-modal feature consistency adjustment: before feature fusion, the feature vectors of different modalities are adjusted to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; multi-head attention mechanism optimization: using a multi-head attention mechanism to increase the processing depth of the model for different feature vectors and improve the accuracy of feature extraction and fusion.

[0169] Optionally, the power dispatch multimodal data processing method also includes: entity label sequence optimization: using a conditional random field model to optimize the entity label sequence output by the graph neural network to ensure the consistency and accuracy of entity recognition results; automatic entity attribute extraction: after entity recognition, automatically extract entity attributes from the multimodal data, including equipment type, operating status, etc., to enrich the entity information of the knowledge graph.

[0170] Optionally, the multimodal data processing method for power dispatch also includes: dynamic updating of entity relationships: dynamically updating entity relationships in the knowledge graph according to real-time changes in the power system to ensure the timeliness and accuracy of the knowledge graph; selection of graph embedding optimization algorithm: selecting a graph embedding optimization algorithm suitable for the power dispatch scenario, and optimizing the embedding representation of entities and relationships according to the algorithm characteristics and task requirements.

[0171] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0172] Embodiments of this disclosure also provide a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the power dispatch multimodal data processing method described in any of the above embodiments when running on a computer or processor.

[0173] Optionally, in this embodiment, the computer program described above may be configured to store a computer program configured to perform the following steps:

[0174] Step S101: Multimodal data acquisition, obtaining diverse and heterogeneous data including text, images, audio and video from the power dispatching system.

[0175] Step S102: Data preprocessing, using natural language processing methods to convert audio data, using OpenCV to preprocess images, and performing frame segmentation on video data.

[0176] Step S103: Multimodal data annotation and sample library construction. The text data is processed using the BIO annotation specification, and entity detection and annotation of the image and video data are performed using CNN to construct a unified knowledge sample library.

[0177] Step S104, Feature Extraction and Fusion: The text features are extracted using the BERT model, the image features are extracted using the improved ResNet model, and the features from different modalities are fused through a cross-modal attention mechanism.

[0178] Step S105, Entity recognition and extraction: Graph Neural Network (GNN) is used for entity recognition, and Conditional Random Field (CRF) model is used to optimize the entity labels to improve the consistency and accuracy of recognition.

[0179] Step S106: Knowledge graph construction. Based on the entity recognition results, establish entity attribute sets and relationships between entities, and optimize the knowledge graph using graph embedding technology.

[0180] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0181] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0182] Optionally, the multimodal data processing method for power dispatch also includes: cross-modal feature consistency adjustment: before feature fusion, the feature vectors of different modalities are adjusted to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; multi-head attention mechanism optimization: using a multi-head attention mechanism to increase the processing depth of the model for different feature vectors and improve the accuracy of feature extraction and fusion.

[0183] Optionally, the power dispatch multimodal data processing method also includes: entity label sequence optimization: using a conditional random field model to optimize the entity label sequence output by the graph neural network to ensure the consistency and accuracy of entity recognition results; automatic entity attribute extraction: after entity recognition, automatically extract entity attributes from the multimodal data, including equipment type, operating status, etc., to enrich the entity information of the knowledge graph.

[0184] Optionally, the multimodal data processing method for power dispatch also includes: dynamic updating of entity relationships: dynamically updating entity relationships in the knowledge graph according to real-time changes in the power system to ensure the timeliness and accuracy of the knowledge graph; selection of graph embedding optimization algorithm: selecting a graph embedding optimization algorithm suitable for the power dispatch scenario, and optimizing the embedding representation of entities and relationships according to the algorithm characteristics and task requirements.

[0185] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0186] Embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the power dispatch multimodal data processing method described in any of the above embodiments.

[0187] Optionally, in this embodiment, the computer program, when executed by the processor, performs the following steps:

[0188] Step S101: Multimodal data acquisition, obtaining diverse and heterogeneous data including text, images, audio and video from the power dispatching system.

[0189] Step S102: Data preprocessing, using natural language processing methods to convert audio data, using OpenCV to preprocess images, and performing frame segmentation on video data.

[0190] Step S103: Multimodal data annotation and sample library construction. The text data is processed using the BIO annotation specification, and entity detection and annotation of the image and video data are performed using CNN to construct a unified knowledge sample library.

[0191] Step S104, Feature Extraction and Fusion: The text features are extracted using the BERT model, the image features are extracted using the improved ResNet model, and the features from different modalities are fused through a cross-modal attention mechanism.

[0192] Step S105, Entity recognition and extraction: Graph Neural Network (GNN) is used for entity recognition, and Conditional Random Field (CRF) model is used to optimize the entity labels to improve the consistency and accuracy of recognition.

[0193] Step S106: Knowledge graph construction. Based on the entity recognition results, establish entity attribute sets and relationships between entities, and optimize the knowledge graph using graph embedding technology.

[0194] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0195] Optionally, the multimodal data processing method for power dispatch also includes: unified annotation of multimodal data: adopting a unified annotation standard for the collected text, image and video data to ensure cross-modal consistency of entity recognition; sample library data augmentation: introducing data augmentation techniques during the sample library construction process, such as rotating, flipping and scaling image data, and performing synonym replacement on text data to improve the diversity of training samples and the generalization ability of the model.

[0196] Optionally, the multimodal data processing method for power dispatch also includes: cross-modal feature consistency adjustment: before feature fusion, the feature vectors of different modalities are adjusted to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; multi-head attention mechanism optimization: using a multi-head attention mechanism to increase the processing depth of the model for different feature vectors and improve the accuracy of feature extraction and fusion.

[0197] Optionally, the power dispatch multimodal data processing method also includes: entity label sequence optimization: using a conditional random field model to optimize the entity label sequence output by the graph neural network to ensure the consistency and accuracy of entity recognition results; automatic entity attribute extraction: after entity recognition, automatically extract entity attributes from the multimodal data, including equipment type, operating status, etc., to enrich the entity information of the knowledge graph.

[0198] Optionally, the multimodal data processing method for power dispatch also includes: dynamic updating of entity relationships: dynamically updating entity relationships in the knowledge graph according to real-time changes in the power system to ensure the timeliness and accuracy of the knowledge graph; selection of graph embedding optimization algorithm: selecting a graph embedding optimization algorithm suitable for the power dispatch scenario, and optimizing the embedding representation of entities and relationships according to the algorithm characteristics and task requirements.

[0199] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0200] In the embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of modules can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through interfaces, or indirect couplings or communication connections between modules, and may be electrical or other forms.

[0201] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0203] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0204] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A method for processing multimodal data in power dispatching, comprising: Multimodal data acquisition: acquiring diverse and heterogeneous data, including text, images, audio, and video, from the power dispatching system; Data preprocessing includes converting audio data using natural language processing methods, preprocessing images using OpenCV, and performing frame segmentation on video data. Multimodal data annotation and sample library construction: The text data is processed using the BIO annotation specification, and CNN is used for entity detection and annotation of the image and video data to build a unified knowledge sample library; Feature extraction and fusion: The BERT model is used to extract the text features, the improved ResNet model is used to extract the image features, and the features from different modalities are fused through a cross-modal attention mechanism; Entity recognition and extraction: Graph Neural Network (GNN) is used for entity recognition, and Conditional Random Field (CRF) model is used to optimize the entity labels to improve the consistency and accuracy of recognition. Knowledge graph construction involves establishing entity attribute sets and relationships between entities based on the entity recognition results, and optimizing the knowledge graph using graph embedding technology.

2. The power dispatch multimodal data processing method according to claim 1, wherein, Also includes: Unified annotation of multimodal data: A unified annotation standard is adopted for the collected text, image and video data to ensure cross-modal consistency of entity recognition; Sample library data augmentation: During the construction of the sample library, data augmentation techniques are introduced, such as rotating, flipping, and scaling image data, and performing synonym replacement on text data, to improve the diversity of training samples and the generalization ability of the model.

3. The power dispatch multimodal data processing method according to claim 1, wherein, Also includes: Unified annotation of multimodal data: A unified annotation standard is adopted for the collected text, image and video data to ensure cross-modal consistency of entity recognition; Sample library data augmentation: During the construction of the sample library, data augmentation techniques are introduced, such as rotating, flipping, and scaling image data, and performing synonym replacement on text data, to improve the diversity of training samples and the generalization ability of the model.

4. The power dispatching multimodal data processing method according to claim 1, wherein, Also includes: Cross-modal feature consistency adjustment: Before feature fusion, the feature vectors of different modalities are adjusted for consistency to ensure that the feature vectors of all modalities are represented in the same lower-dimensional space, which facilitates the subsequent cross-modal attention mechanism processing; Multi-head attention mechanism optimization: Using a multi-head attention mechanism increases the processing depth of the model for different feature vectors, thereby improving the accuracy of feature extraction and fusion.

5. The power dispatching multimodal data processing method according to claim 1, wherein, Also includes: Entity label sequence optimization: The entity label sequence output by the graph neural network is optimized using a conditional random field model to ensure the consistency and accuracy of entity recognition results; Automatic entity attribute extraction: After entity recognition, the system automatically extracts entity attributes from multimodal data, including device type, operating status, etc., to enrich the entity information of the knowledge graph.

6. The power dispatching multimodal data processing method according to claim 1, wherein, Also includes: Dynamic updates of entity relationships: Based on real-time changes in the power system, the entity relationships in the knowledge graph are dynamically updated to ensure the timeliness and accuracy of the knowledge graph; Graph embedding optimization algorithm selection: Select a graph embedding optimization algorithm suitable for the power dispatching scenario, and optimize the embedding representation of entities and relations based on the algorithm characteristics and task requirements.

7. A power dispatching multimodal data processing system, wherein, include: The acquisition module is configured for multimodal data acquisition, obtaining diverse and heterogeneous data, including text, images, audio, and video, from the power dispatching system; The processing module is set up for data preprocessing, using natural language processing methods to convert audio data, using OpenCV for image preprocessing, and performing frame segmentation on video data. The first construction module is set up for multimodal data annotation and sample library construction. It uses the BIO annotation specification to process the text data, uses CNN to perform entity detection and annotation of the image and video data, and builds a unified knowledge sample library. The fusion module is configured for feature extraction and fusion. It uses the BERT model to extract text features, uses the improved ResNet model to extract image features, and fuses features from different modalities through a cross-modal attention mechanism. The recognition module is configured for entity recognition and extraction. It uses a graph neural network (GNN) for entity recognition and a conditional random field (CRF) model to optimize the entity labels, thereby improving the consistency and accuracy of recognition. The second construction module is set up for knowledge graph construction. It establishes entity attribute sets and relationships between entities based on the entity recognition results, and optimizes the knowledge graph using graph embedding technology.

8. An electronic device comprising a memory and a processor, wherein, The memory stores a computer program, and the processor is configured to run the computer program to perform the power dispatch multimodal data processing method as described in any one of claims 1 to 6.

9. A non-volatile storage medium, wherein, The non-volatile storage medium stores a computer program, wherein the computer program is configured to execute the power dispatch multimodal data processing method as described in any one of claims 1 to 6 when running on a computer or processor.

10. A computer program product, wherein, It includes a computer program that, when executed by a processor, implements the power dispatch multimodal data processing method according to any one of claims 1 to 6.

Images