277 results about "Data repair" patented technology

The invention discloses an AI-based data asset quality assessment and automatic repair method and system, and the method comprises the steps: receiving a to-be-processed target data asset, inputting the target data asset into a multi-dimensional joint quality scoring engine, and generating a comprehensive quality assessment report containing an integrity defect map, a distribution abnormality thermodynamic diagram and a semantic conflict link; based on the comprehensive quality evaluation report, utilizing a rule and learning fused strategy generator to analyze repair requirements and constraint conditions contained in a defect map, an abnormal thermodynamic diagram and a conflict link, and dynamically generating repair operation instruction sets and priority sequences for different types of defects; and inputting the target data assets, the repair operation instruction set and the priority sequence into a multi-expert collaborative repair execution module, repairing defects in the target data assets according to the priority sequence, and outputting the repaired data assets. According to the embodiment of the invention, comprehensive and accurate data quality evaluation can be realized, and the automation level and accuracy of data restoration are improved.

The invention discloses an enterprise data link management and value management method and system, and the method comprises the steps: carrying out the real-time capturing of multi-source heterogeneous data of all business systems in an enterprise through a distributed data collection engine, and recognizing the types of original data in different formats through a preset data source adapter; if structured data is detected, a relational database connector is adopted for extraction, and if the structured data is recognized as unstructured data, a document analysis module is started for content extraction, and an initial data set containing metadata tags is obtained; performing format conversion and field mapping on the initial data set according to a pre-established data standardization rule base, eliminating duplicate records and abnormal values through a data cleaning algorithm, performing automatic evaluation on a data quality grade by adopting a naive Bayes classifier, if a data quality score is lower than a preset threshold value, triggering a data recovery process, and if the data quality score is lower than the preset threshold value, performing data recovery. And standardized data meeting a unified standard is obtained. The normativity and value utilization efficiency of data management are effectively improved.

The invention discloses a data processing system based on an artificial intelligence algorithm, and relates to the technical field of artificial intelligence and data processing, and the system comprises a data input interface which is used for receiving a multi-source heterogeneous data stream; and the preprocessing engine is connected with the data input interface and comprises a dynamic metadata sensing unit, an incremental quality evaluation unit, a bidirectional verification self-repairing unit and a closed-loop feedback optimization unit. According to the data processing system based on the artificial intelligence algorithm, through dynamic metadata perception and an incremental quality evaluation mechanism, the manual intervention requirement of a multi-source heterogeneous data preprocessing stage is reduced, and the problem of adaptation stiffness caused by the fact that a traditional method depends on a static rule is solved; by combining the bidirectional verification design of the business rule and the model feedback, the dual reliability of the data recovery strategy in logic rationality and algorithm compatibility is ensured, the risk of overfitting or scene mismatching caused by a single verification mechanism is avoided, and the input data quality and decision accuracy of a downstream artificial intelligence model are improved.

Certain example embodiments relate to dashboards that help streamline and automate data quality management processes used with machine learning (ML) models and ML-enabled technology. A clean dataset is initialized from a dirty dataset. A search space is the set of all possible combinations of available error detection algorithms and data repair algorithms. A scoring function measures performance of a given error detection algorithm and data repair algorithm combination on the clean dataset. An ML model is trained using the clean dataset. Best error detection and data repair algorithms are selected, based on an optimization on the set of all possible combinations, and the defined scoring function. The selected best error detection algorithm is applied to the clean dataset, and a repaired dataset is generated using the selected best repair algorithm. The clean dataset is set to the repaired dataset. This procedure is repeated until a condition is met.

The invention discloses a plant protection field literature information batch structured extraction method based on an AI large model, and belongs to the technical field of computer data processing and artificial intelligence application. According to the method, a PDF format literature is converted into a Markdown format text through a PDF literature self-adaptive preprocessing module and a Markdown conversion module; then, a double-strategy self-adaptive AI extraction module is adopted, different modes are adopted for processing according to a text length threshold value, and JSON data are extracted in combination with a JSON format data restoration strategy; then, the CPU intensive conversion task and the I / O intensive AI analysis task are processed in parallel through a two-stage parallel scheduling module, and a structured database file is generated through aggregation according to a predefined mapping rule through a multi-dimensional aggregation export module. According to the method, end-to-end automation from literature acquisition to structured data output is realized, and the problem of low extraction efficiency of manual literature reading information is solved.

The invention discloses a data processing method and device, electronic equipment, a storage medium and a product. The method comprises the steps of obtaining a disk dynamic mapping request; in response to the disk dynamic mapping request, a data repair migration task is generated, and the data repair migration task is used for triggering disk dynamic mapping and data repair migration processes of the erasure code redundant group; and executing the data restoration migration task.

The invention provides a hydroelectric generating set abnormal data identification and missing data restoration method which is used for monitoring data of a hydroelectric generating set, and the identification method selects a K-means clustering algorithm to detect abnormal values of the monitoring data according to time sequence delay and mode similarity characteristics of the monitoring data. According to the repairing method, a time sequence data repairing model based on a self-attention mechanism is constructed, and the time sequence data repairing model is used as a generator and is matched with a discriminator and a loss function to form a self-attention generative adversarial network SA-GAN for filling the missing monitoring data of the hydroelectric generating set; according to the method, the operation state of the hydroelectric generating set can be analyzed more accurately, more reliable data support is provided for operation and maintenance personnel, and then the operation efficiency and safety of the hydroelectric generating set are improved.

The invention discloses an electric meter historical data restoration method based on a multi-dimensional incidence relation, and relates to the technical field of electric power data management and intelligent restoration, the accuracy and traceability of data restoration are realized by establishing a multi-source exception type library and an element time-varying degradation model, and the accuracy and traceability of data restoration are improved for multi-source data conflicts. The method comprises the following steps: pre-defining a plurality of abnormal mode combinations and corresponding repair strategies, positioning a fault source through mode matching of a real-time index and an abnormal type library, selecting reference data in combination with a historical average deviation rate of a target ammeter and an associated ammeter, avoiding deviation caused by single judgment, and for nonlinear degradation, firstly determining fault elements of a shunt and an ADC chip, a resistance degradation model and a conversion error model are constructed, a theoretical correction value is calculated, a result is optimized through a machine learning model trained by historical normal data, meanwhile, original readings, model parameters and calculation bases are stored in an associated mode, a traceable log is formed, and the repair accuracy is guaranteed.

The invention relates to the technical field of data analysis and processing, in particular to an automatic data standardization management method based on a large language model, a medium and equipment, and the method comprises the following steps: a data receiving step: receiving data from various data sources; a data source analysis step: identifying field semantics and standardization requirements; a standardized rule generation step: based on the semantic understanding result of the large language model, automatically generating a standardized rule suitable for a corresponding data source; a standardization rule application and optimization step: performing application and dynamic optimization; an intelligent monitoring and verification step of data quality: performing real-time quality detection of the data by using a large language model; and a data restoration and suggestion step: automatically deducing a correct value or giving a restoration suggestion when the data does not meet the standardization requirement. According to the method, the problems of non-uniform standards, high manual rule maintenance cost and lack of dynamic adaptivity caused by inconsistent formats and semantics of the multi-source data are solved.

The invention provides an offshore wind turbine abnormal data recovery and risk early warning method and system, and the method comprises the steps: receiving the multi-modal monitoring data of an offshore wind turbine in real time, the multi-modal monitoring data comprising time domain sensor data, frequency spectrum images and environmental parameters; inputting the multi-modal monitoring data into a pre-trained visual language large model, and inputting the frequency spectrum image into a visual encoder of the visual language large model to extract frequency domain features; time domain sensor data and environmental parameters are input into a language encoder of the visual language large model to generate time sequence features and semantic features; and performing cross-modal fusion on the frequency domain features, the time sequence features and the semantic features through a cross-modal fusion module of the visual language large model to obtain cross-modal fusion features, and synchronously executing abnormal data recovery and risk early warning based on the cross-modal fusion features to obtain recovered complete monitoring data and a risk early warning result. According to the method, time domain and frequency domain data are combined, and the multi-mode large model is utilized to realize high-precision data restoration and intelligent risk early warning.

The invention provides an intelligent denitration control system based on multi-model fusion and self-learning, and belongs to the technical field of industrial denitration control, and the method comprises the following steps: collecting three-dimensional data of a boiler side, a combustion side and a denitration side in a DCS / SIS system in real time through an OPC protocol; s2, an intelligent data preprocessing step, wherein the step comprises the links of anomaly detection, data restoration and dynamic normalization; s3, a spatio-temporal feature engineering step, wherein the step comprises a time delay analysis link, a feature derivation link and a feature screening link; s4, a multi-model dynamic fusion step, wherein the step comprises a base model group and a fusion mechanism link; and S5, an online self-learning optimization step, the step comprises incremental learning, parameter evolution and knowledge distillation links, and aiming at the problem that algorithm optimization adjustment cannot be carried out aiming at the environmental change of the denitration process, self-adaptive evolution of control parameters is realized by constructing a'prediction-control-feedback 'complete chain.

The invention discloses a sewage treatment time series data missing value interpolation method based on a random configuration network. The method comprises the following steps: collecting water quality data in a sewage treatment process, and preprocessing the water quality data to obtain univariate time series data; performing seasonal trend decomposition on the single-variable time series data to obtain a seasonal component, a trend component and a residual component; respectively performing component interpolation on the season component, the trend component and the residual component; and performing addition reconstruction on the seasonal component, the trend component and the residual component after interpolation to obtain complete sewage treatment process monitoring data. According to the sewage treatment time sequence data missing value interpolation method based on the random configuration network, original data is decomposed into season, trend and residual components, interpolation is conducted on all the components through self-similarity decomposition and a random configuration network model, finally a complete sequence is reconstructed, the periodicity and trend characteristics of missing data can be effectively recovered, and the accuracy of the missing data is improved. And the restoration precision and the application reliability of the sewage treatment monitoring data are improved.

The invention relates to a data recovery method and device, computer equipment, a storage medium and a program product. The method comprises the steps that a current repairing task of a hybrid tree is determined, the hybrid tree is a hybrid storage structure of a log merging tree LSM tree and a B tree, the hybrid tree comprises multiple layers, the multiple layers comprise the first layer of the LSM tree and all layers of the B tree, ordered character string tables of the first layer of the LSM tree are merged and sunk into the B tree, and each node in the B tree stores at least one ordered character string table obtained through merging and sinking; under the condition that the current repair task indicates that the repaired damaged data storage object belongs to the B tree, obtaining a B tree index of the damaged data storage object from the current repair task; searching a damaged data storage object from the mixed tree according to the B tree index, wherein the damaged data storage object is an ordered character string table or a node; and under the condition that the damaged data storage object is found, repairing the damaged data storage object. By adopting the method, the data recovery efficiency can be improved.

The invention provides a sensor data recovery method and system based on mask perception space-time modeling, and belongs to the technical field of Internet of Things and intelligent sensing data processing. The method comprises the following steps: modeling a missing position into a trainable potential vector representation through an adaptive missing representation module, and avoiding noise introduced by traditional zero filling or mean filling; a missing perception space-time decoding module is adopted, and a dynamic weight distribution mechanism of mask constraint is combined in the decoding process, so that information is effectively prevented from being excessively smooth; designing a space-time dual-channel feature aggregation unit, and capturing spatial dependence and time sequence dependence between the sensors at the same time; and finally realizing accurate completion of a large-scale space-time sensor matrix through an output recovery unit. According to the method, the accuracy and robustness of sensor data restoration can be remarkably improved, the reliability of subsequent monitoring, prediction and anomaly detection is enhanced, and the method has wide engineering application value.

The invention discloses an automatic data cleaning and anomaly recognition method for financial auditing, and relates to the technical field of financial auditing. Extracting fields from the multi-source heterogeneous data by using a regular expression, and unifying the format through semantic mapping; repairing numeric missing data by using LSTM, and checking logic errors by using a rule engine; identifying abnormal transactions based on an isolated forest algorithm in combination with business rules; constructing a risk assessment matrix to divide risk levels; dynamic knowledge graph analysis implicit association can be constructed, an incremental learning optimization model is utilized, and an auditing evidence chain and block chain evidence storage are recorded. According to the invention, automatic processing of financial audit data is realized, the data recovery success rate is improved, and the anomaly identification accuracy is improved; hidden risks are mined through the knowledge graph, audit traceability is guaranteed through block chain evidence storage, manual annotation cost is reduced by 40%, and audit efficiency and reliability are improved.

The invention discloses a BDDR backdoor detection and data recovery method and system oriented to a large model, and belongs to the field of backdoor defense. Comprising the following steps: constructing a knowledge distillation architecture under federal learning, including an edge server and a plurality of clients, and obtaining distillation data; inputting the distillation data into a randomly initialized model for training, recording the loss change of each batch of data, and screening out abnormal batches to form a backdoor data set; the edge server initializes two independent models, respectively uses a distillation data set and a backdoor data set for training, and guides learning of backdoor features; using probability distribution to calculate and correct a backdoor label, generating a clean data set by adding noise, finely adjusting a large model, detecting residual backdoor feature intensity, and adjusting probability distribution calculation parameters to further weaken backdoor features according to the residual backdoor feature intensity so as to obtain a final repaired data set; while the generalization ability of the large model is improved, backdoor attacks can be effectively identified and defended, the data privacy of the client is protected, and the model security is ensured.

The invention relates to a fan assembly process missing data filling method based on semi-supervised learning, and belongs to the technical field of industrial data missing repair. Aiming at the technical problems that the traditional linear interpolation and supervised learning are difficult to process the missing of wind power assembly multi-source heterogeneous data, the dynamic working condition adaptability is poor and the annotation dependence is high, a feature extraction method of combining space-time tensor modeling with a bidirectional space-time attention network is provided, and the reliability of three-dimensional space-time tensor quantitative data is constructed; designing an adversarial domain adaptation mechanism to realize feature space alignment of labeled and unlabeled data; a dynamic confidence threshold strategy is adopted to screen high-credibility samples, and a collaborative loss function optimization model is constructed; the method has the technical effects that the data filling precision and the process conformity under the complex working condition are remarkably improved, the data labeling requirement is reduced, flexible production is adapted, the cross-production-line data island problem is solved, the real-time repair efficiency is optimized, assembly quality tracing and process optimization are enabled, and reliable data support is provided for intelligent wind power manufacturing.

The embodiment of the invention provides a data recovery method and device, a storage medium and an electronic device.The method comprises the steps that abnormal second data in first data is determined according to a data rule engine, and a data rule applied in the data rule engine is configured according to a data service scene corresponding to the first data; the second data is restored through a preset processing mode, the preset processing mode comprises a data prediction generation algorithm, and the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring mode. By the adoption of the technical scheme, the problem that the data restoration effect is poor due to the fact that a data restoration method in the related technology cannot be updated in real time along with changes of the restored data is solved.

The invention relates to a local-global association learning-based multi-modal power data restoration method, which comprises the steps of obtaining initial data and performing preprocessing to obtain a power data abnormal value, the initial data comprising power operation data, external association data and multi-modal data; based on the preprocessed power operation data and the power data abnormal value, local association learning is carried out, local data association is mined, and a local restoration scheme of the power data abnormal value is obtained; performing global association learning based on the preprocessed initial data and the abnormal value of the power data, mining a cross-time and cross-source global association rule, and obtaining a reasonable correction range of the abnormal value of the power data; and generating a power operation data restoration result in combination with the local restoration scheme and the reasonable correction range. Compared with the prior art, the method has the advantages that the data exception can be accurately recognized and repaired, and the data repair accuracy is greatly improved.

The invention relates to the technical field of wind energy development and utilization, in particular to a multi-source meteorological data dynamic fusion wind power plant simulation method, which comprises the following steps of: accessing satellite remote sensing, a ground observation station, numerical forecasting and wind power plant historical operation data, and outputting standardized meteorological data through format analysis, standardized processing and cleaning; a credibility score is calculated based on multi-dimensional indexes such as data integrity and time consistency, and abnormal data restoration or standby data source switching is achieved; in combination with wind power plant DEM topographic features and a self-adaptive sliding window, the weight of a data source is adjusted in real time through reinforcement learning, and a dynamic weight value is generated; an improved Windpowerlib framework adapts to fusion data, fan parameters are dynamically configured, and an initial simulation result is generated; and collecting real-time data of the wind power plant, and correcting model parameters by using Kalman filtering. The wind power plant simulation precision and timeliness can be improved, the data utilization rate and the model adaptability are enhanced, and the deviation between the simulation result and the actual working condition is reduced.

The invention discloses a traffic flow state data time gathering method based on Gaussian mixture probability, and belongs to the technical field of calculation, reckoning or counting. The method comprises the following steps: acquiring traffic flow state data and vehicle trajectory data of each lane in a target road section; carrying out probability diagnosis on the traffic flow data by adopting a Gaussian mixture model, and identifying and deleting abnormal points; introducing a model based on Transform and graph neural network fusion, and realizing space-time repair of abnormal data through joint modeling of an attention mechanism and a space-time topology dependency relationship; and carrying out time window collection on the repaired data based on wave walking and stopping prediction, and calculating a traffic flow rate, an average speed and an occupancy rate. According to the method, the joint characteristics of the traffic flow state data and the trajectory data are fully utilized, meanwhile, the abnormal data repairing precision and the space-time consistency are improved, adaptive collection of the traffic operation dynamic characteristics is achieved by introducing walking and stopping wave prediction, and the robustness and reliability of traffic flow data processing are remarkably improved.

The invention relates to the technical field of data restoration, in particular to a traffic data space-time restoration method based on time domain-frequency domain evolution, which can accurately capture dynamic characteristics of a traffic situation on different time scales through the method based on time domain-frequency domain evolution, and improves the modeling capability for a complex traffic situation. According to the method, the inverse Fourier transform technology and the wavelet transform technology are utilized, the fitting capacity for actual time fluctuation can be guaranteed during time signal reconstruction, the diversity characteristics of the traffic situation can be reserved, and the quality of time signal reconstruction under emergencies is effectively improved. According to the method, the GAT model and the Transform model are combined, missing data are repaired in the space-time dimension, the space-time interaction characteristics of the traffic flow can be fully utilized, missing area data can be accurately deduced, time and space dynamic characteristics are captured, the accuracy and integrity of a repair result are further optimized, and a more reliable basis is provided for traffic flow prediction and management.

This application discloses a distributed new energy data scheduling and processing method based on cloud-edge-end collaboration, which relates to the field of distributed new energy data analysis technology. Specifically, the application discloses the following steps: the cloud computing center outputs a task scheduling strategy adapted to the task processing capabilities of each edge node to the terminal device based on edge node information, terminal device information, and the edge node task scheduling model; the terminal device uploads the distributed new energy operation data to the corresponding edge node according to the task scheduling strategy and encryption strategy; the edge node performs abnormal data identification and correction on the distributed new energy data based on the abnormal data detection model and the abnormal data repair model. The beneficial effects of this application are: reducing transmission time, improving data processing efficiency, and enhancing the timeliness and accuracy of data output.

The invention relates to the technical field of data processing, and discloses a data online modeling and data management method for dimension modeling. The method comprises the following steps: performing dimension characteristic baseline acquisition on a multi-source heterogeneous data stream to generate a dimension attribute characteristic set; performing dimension hierarchical modeling based on the feature set, and outputting a result containing an atomic layer, an aggregation layer and an application layer; calculating a dimension correlation degree according to the topological relation of the service entities, and generating a correlation constraint parameter; detecting dynamic modeling conflicts based on the parameters, and outputting a conflict dimension identification set; distributing a query direction optimization weight according to the identifier set to obtain a dimension query optimization strategy; incremental data loading is controlled based on a strategy, and an incremental loading sequence divided according to time windows is generated; performing closed-loop consistency verification on the sequence, and outputting a verification result set containing data integrity and association consistency indexes; recording a modeling state snapshot according to the result set, and generating a snapshot with a version mark; and analyzing the abnormal backtracking path based on the snapshot, and outputting a dimension data recovery path instruction.

This invention discloses an artificial intelligence-based method for managing and controlling power grid water dispatch data, specifically relating to the field of power grid water dispatch data management technology. The method involves collecting multi-source heterogeneous water dispatch data and its metadata to construct a unified format dataset; generating a credibility score based on metadata and data features to form a credibility matrix; constructing a multi-dimensional quality assessment tensor and extracting the dominant factor subspace; inputting this tensor into a clustering perceptual neural network to identify abnormal data, outputting an abnormal dataset and abnormal labels; combining a large model enhanced by RAG retrieval with historical similar data to complete abnormal data repair; constructing a data quality improvement report using the difference vectors before and after repair and label information; and finally pushing the report to a data management platform and generating a traceable rectification work order. This method achieves automatic identification, intelligent repair, and feedback tracking of power grid water dispatch data, improving data credibility, governance efficiency, and security, and is suitable for data management needs in complex scenarios under new power systems.

The application provides a ground target threat evaluation method in an uncertain environment, and the target feature data is repaired by using an improved autoregressive model for the information missing problem of the ground target according to the acquired target situation information; the target threat data sample is generated by using the explainability of the Bayesian network for the problems existing in the current threat evaluation method, and the target threat evaluation model is established by using the nonlinear fitting ability of the BP neural network, so that the final ground target threat level evaluation result is output. The application can efficiently process data missing, solve the problems of threat evaluation sample shortage and model precision, and also integrates the advantages of the model, considers the performance and explainability, is more suitable for the time sequence characteristics and dynamic changes of the ground target data compared with the traditional method, improves the accuracy and robustness of data repair, and provides more complete and reliable basic data for subsequent evaluation.

This invention discloses a method, device, system, and storage medium for longitudinal protection of distribution networks, including: 1) extraction of key response data strongly correlated with line protection and repair of abnormal data; 2) a longitudinal protection method based on Hellinger distance; and 3) a distribution network longitudinal protection method based on abnormal data repair and Hellinger distance. This application uses cubic trigonometric Hermite interpolation with shape parameters to identify and repair fault current data, effectively solving the problem of protection maloperation or failure to operate due to data loss or distortion, and improving the data reliability and adaptability of protection under complex operating conditions.

The invention discloses a power grid fault-oriented multi-level spatio-temporal data processing and visualization method, comprising spatio-temporal reference unification, including sliding window adaptive interpolation and interpolation result optimization; performing spatial reference normalization, including spatial quantization alignment and power grid-meteorological coupling modeling; data governance, redundancy elimination and missing repair, including dynamic time warping de-duplication, principal component analysis dimension reduction and missing data repair; the visual architecture design comprises a physical power grid layer, an alarm signal layer and a disposal process layer. Through the spatio-temporal data fusion technology, the operation state and fault characteristics of the power grid equipment are accurately captured, and timely and accurate fault diagnosis support is provided; a hierarchical visual architecture is designed, so that a user can go deep into a microscopic equipment level from a macroscopic power grid state layer by layer, fault positioning and analysis are helped to be more accurate and efficient, understanding of power grid dispatchers on fault conditions is enhanced, and rapid response and decision making are achieved.

The application discloses a data repairing method, comprising: segmenting an RGBW image according to point cloud data of a workpiece to be measured, the RGBW image and a preset segmentation algorithm, determining a plurality of target regions and a data missing region; fitting a zero plane according to the point cloud data and a least square method, and converting the point cloud data to the zero plane to obtain converted point cloud data; determining point cloud data located around each point in the data missing region in the converted point cloud data, and determining an effective value of each point; repairing the data missing region according to a positional relationship between the data missing region and the plurality of target regions, an angle value and the effective value of each point in the data missing region, and obtaining repaired point cloud data of the workpiece to be measured. Thus, corresponding repairing modes are collected for data missing regions located in different target regions, so that the repaired point cloud data of the workpiece to be measured can better restore the actual height of the workpiece to be measured.

This invention relates to the field of data monitoring technology, and more particularly to an enterprise data monitoring system based on big data analytics, used to solve the problems of incomplete, inaccurate, and duplicated data from multiple channels. It includes a data acquisition module to determine the data missing rate and data consistency; a data diagnosis module to determine data categories to differentiate data analysis methods; a data analysis module to calculate data anomaly characteristics, classify data anomaly trends to determine data repair methods, or mark usable data; a data repair module, for data with strong anomaly trends, dividing the data into several anomalous text data frames, predicting the anomalous text data frame groups, and verifying the predicted anomalous text data frame groups; for data with weak anomaly trends, using a trained convolutional neural network for repair, determining successfully replaced data; and a data processing module to determine data availability and mark usable data.