216 results about "Data lineage" patented technology

The invention provides an intelligent data quality monitoring method and system. The method comprises the following steps: performing data processing on multi-source heterogeneous data by utilizing a unified data model; inputting historical data as a training set into the pre-training model for training optimization to obtain an optimized pre-training model, and performing threshold determination on the standardized distribution data stream by using the optimized pre-training model in combination with an adaptive threshold module; inputting the processed abnormal event list into a context awareness and dynamic priority management module to carry out de-duplication optimization transmission; carrying out root analysis on the abnormal event list after the de-weighting optimization by utilizing a root analysis tool; generating a repair strategy for the root analysis result by using the assistance of a repair suggestion engine; and inputting the repair strategy and the visual report into a cross-platform monitoring interface, and generating a repair suggestion and an early warning notification. According to the method, the data consanguinity map is constructed based on the graph database, minute-level traceability of abnormal events is achieved, and the troubleshooting time is shortened by 50%.

The invention relates to the technical field of data analysis, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a data consanguinity map processing method, device, equipment and medium, and the method comprises the steps: collecting metadata change information and data operation event information, and building a mixed consanguinity map model; monitoring a metadata change event and a data operation event, and triggering incremental updating based on an event type; and performing data change influence analysis based on the updated blood relationship map model, generating an analysis result, and outputting the mixed blood relationship map model and the data change influence analysis result. According to the invention, by collecting metadata, incrementally updating the blood relationship map and carrying out data change influence analysis, full-link data tracing and influence analysis are realized, the problems of blood relationship breakage and insufficient real-time data tracing in traditional data blood relationship analysis are solved, and the accuracy of data tracing and the timeliness of business decision are improved.

In one implementation, a device may maintain access permissions that control whether a user is allowed to access a particular document. The device may match a plurality of document chunks from a retrieval augmented generation system to a prompt issued by the user for input to a language model. The device may form, based on the access permissions, a modified set of document chunks by excluding a particular document chunk from the plurality of document chunks based on the particular document chunk having a data lineage from the particular document. The device may augment the prompt using the modified set of document chunks prior to input to the language model.

Systems and methods of the present disclosure enable anomaly detection based on data lineage by using at least one processor to receive initial lineage records that record initial changes in data items at each data processor of a lineage stream. The processor(s) generates a historical distribution as a data processor baseline for a particular data processor based on each change in each data item at the particular data processor. For lineage records in a subsequent time period, the processor(s) identifies subsequent changes in data items at the particular data processor and generates a real-time dynamic distribution to represent a current data processor behavior for the particular data processor based on each change for the subsequent lineage records. The processor(s) determines a deviation between the data processor baseline and the current data processor behavior based on the historical distribution and the real-time dynamic distribution to identify a data processor anomaly.

Systems and methods for providing a data lineage are described. According to some examples, a method may include receiving a source dataset corresponding to a use case scenario. The method may further include classifying the source dataset based on the use case scenario to create a classified dataset and then applying a set of data processing rules to the classified dataset to create a curated dataset. The method may further include receiving a reporting data object indicating a relationship between the source dataset and the curated dataset. The relationship can describe a data lineage of the source dataset. The method may further include outputting display data associated with the reporting data object.

The invention discloses an auditing and supervising system and method for real-time traceability of multi-source data, and relates to the technical field of data governance, and the method comprises the steps: when data is generated, automatically collecting original feature information of the data, writing the original feature information into a block chain, generating a unique data consanguinity identifier, and building a data consanguinity index table; during data circulation, recording node identifiers, timestamps and operation types in real time, acquiring attenuation values from the attenuation coefficient comparison table, calculating current credibility coefficients, and chaining the current credibility coefficients as new nodes to form chained association; the data integrity is periodically verified, when it is detected that the data quality is abnormal, a complete blood relationship chain corresponding to the abnormal data is called, and the attenuation value and the offset cumulant of each link are associated; and screening links of which the decreasing amplitude of the credibility is greater than a threshold value, calling an uplink record of a maximum attenuation point in combination with a feature offset jump point, and positioning a specific operation behavior. According to the invention, the maximum attenuation point and the specific operation behavior of the abnormal data are accurately positioned, and the auditing efficiency is improved.

The invention discloses a data acquisition method and device for an energy management system, equipment and a medium. The energy management system is used for managing multiple power plants. The method comprises the following steps: creating a to-be-constructed big data platform which comprises a data acquisition layer, a data processing layer and a data storage layer; deploying a multi-protocol adapter cluster on a data acquisition layer; constructing a data standardization pipeline in a data processing layer, and generating a multi-level security label in combination with a data source security level, a service association degree and an updating frequency; a hierarchical storage framework is designed in a data storage layer and used for constructing a redundant index table crossing storage nodes and a data blood relationship graph based on a graph database; and deploying a dynamic security gateway between the data acquisition layer and the data processing layer, dynamically selecting a corresponding encryption algorithm according to the data security label, and completing the construction of a big data platform, so as to complete the data acquisition of the plurality of power plants based on the big data platform. And high-reliability, traceable and safe data support is provided for cooperative management of multiple power plants.

The invention provides a data governance method and device based on data consanguinity and electronic equipment, and relates to the field of data governance. The method comprises the following steps: acquiring metadata of target data corresponding to a target governance demand; constructing a data blood relationship map of the target data according to the metadata; calculating a data quality propagation score, a data blood relationship path complexity, a data global influence score, a data leakage risk score and a data use efficiency score of the target data through the data blood relationship map; generating a data governance comprehensive score based on the data quality propagation score, the data consanguinity path complexity, the data global influence score, the data leakage risk score and the data use efficiency score; and performing data governance on the target data according to the data governance comprehensive score. By implementing the technical scheme provided by the invention, accurate treatment on the data is facilitated.

The invention relates to the field of cold and hot data storage, in particular to a cold and hot data storage optimization method based on a large model, which comprises a business integration module, a data dependence module, a mode updating module, a cold and hot partitioning module and a file merging module, and is characterized in that the business integration module is used for collecting business data to form a training data set; the data dependency module is used for fitting business data to obtain a data consanguinity model, the mode updating module is used for predicting an access mode, the cold and hot partitioning module is used for calculating the heat of the data and performing dynamic scheduling, and the file merging module is used for automatically merging small files. A data storage structure is standardized, storage space is reduced, more efficient and more accurate cold and hot data classification can be achieved, the resource utilization rate of data storage hardware is improved, hot data response delay is reduced, the system storage utilization rate is remarkably improved, and the storage efficiency is improved.

The invention discloses a large model-knowledge graph driven data blood relationship traceability reasoning method and system. According to the method, through a closed-loop framework of KG-driven dynamic decomposition, small-model lightweight reasoning and visual credibility verification, the defects are systematically solved, and an interpretable, low-cost and easy-to-expand data consanguinity depth traceability scheme is provided for scenes with high credibility requirements such as data governance, compliance auditing and root cause analysis. According to the system, aiming at complex problem requirements, after a complex problem is input, the problems of no interpretation, low reliability and the like in data blood relationship traceability are solved through two modules, namely problem reasoning and verification presentation. The modular design not only conforms to the technical decoupling principle, but also facilitates independent optimization, and is suitable for medical treatment, scientific research and other scenes with high requirements for interpretability.

The invention provides an ecological environment field-oriented data quality detection method and system based on a large model and a storage medium, and the method comprises the steps: 1, connecting a database, and constructing a plurality of knowledge bases according to the planning of a data governance library and a data dictionary; 2, formulating a general rule, an ecological environment field special rule and a user-defined rule; 3, checking whether the rule is reasonable and effective or not through a prompt template special for a large model in combination with the data consanguinity and the data rule; 4, creating a corresponding scheduling task list according to the current effective data quality rule and the updating frequency; 5, checking the quality problem of the collected data through a prompt template special for a large model, and outputting an analysis report; and step 6, generating improvement suggestions for the problem points found in the step 5 through a large model special prompt template in combination with the situation and reason of the problem. The method has the beneficial effect that the scientificity and effectiveness of rule checking are improved.

The invention belongs to the field of big data, and relates to a data consanguinity analysis visualization method, which comprises the following steps of: obtaining metadata of target data from a plurality of data sources, and determining a metadata type and a metadata attribute of the metadata; obtaining a blood relationship model of the target data; based on the blood relationship model, the metadata type and the metadata attribute, performing data blood relationship analysis on the target data by adopting a preset graph algorithm to obtain a blood relationship analysis result of the target data; and based on the blood relationship analysis result, establishing a blood relationship graph of the target data, and displaying the blood relationship graph in a preset interactive interface. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the metadata can be stored in a block chain. The method can be applied to the fields of financial insurance, medical health management and the like, and the blood relationship graph of the data can be efficiently and accurately established and displayed.

The invention relates to a user operation risk dynamic monitoring method and device based on data consanguinity and a medium, and the method comprises the following steps: obtaining operation behaviors of all users in real time, and extracting a segment of continuous operation behaviors of each user as a group of behavior fragments with semantic relevance; a plurality of continuous behavior segments of each user are matched with a preset behavior template, so that a plurality of intermediate behavior blocks are obtained through dimension reduction; according to a time sequence, traversing and extracting a predetermined number of intermediate behavior blocks from the plurality of intermediate behavior blocks, and respectively combining to generate a behavior fragment sequence; and respectively inputting the behavior fragment sequences into a pre-trained neural network model to obtain a risk classification result or a risk probability distribution result corresponding to each behavior fragment sequence. The method has the advantages that each operation behavior can be subjected to structured expression, the potential behavior chain can be reconstructed through the combination strategy, and the high-risk behavior path hidden in the data noise is recognized by means of the deep model.

The present invention provides an intelligent data quality monitoring method and system. The method includes: performing data processing on multi-source heterogeneous data using a unified data model; using historical data as a training set to input into a pre-trained model for training and optimization to obtain an optimized pre-trained model, and performing threshold determination on the standardized distributed data stream using the optimized pre-trained model in combination with an adaptive threshold module; inputting the processed list of abnormal events into a context awareness and dynamic priority management module for deduplication and optimized transmission; performing root cause analysis on the deduplicated and optimized list of abnormal events using a root cause analysis tool; using a repair suggestion engine to assist in generating a repair strategy based on the root cause analysis result; and inputting the repair strategy and a visualization report into a cross-platform monitoring interface to generate repair suggestions and warning notifications. The present invention constructs a data lineage graph based on a graph database, realizes minute-level traceability of abnormal events, and shortens the fault troubleshooting time by 50%.

The invention provides a blood relationship analysis method based on a data dependency relationship, electronic equipment and a medium, and relates to the technical field of data blood relationship analys.The method comprises the steps that an SQL statement vector corresponding to each SQL statement is generated according to all table names and field names corresponding to each SQL statement; obtaining an SQL statement vector list A; all SQL statement vectors in A are clustered to obtain a cluster list B; analyzing each SQL statement in the Bj to extract a corresponding table name, a field name and an operation type, and establishing a field-level blood relationship graph Xj between the table names and the field names of the SQL statements in the Bj; establishing a cross-cluster blood relationship mapping table QR corresponding to the B; fusing the clusters in the B to obtain a global blood relationship analysis graph QW; displaying the QW in a layered interactive view; according to the method, the bottlenecks of the traditional blood relationship analysis method in accuracy, efficiency and availability are solved, and efficient and reliable technical support is provided for large-scale data management.

The invention provides a data blood relationship tracking method and device, computer equipment and a storage medium. The method comprises the following steps: respectively obtaining data source metadata, ETL task metadata and index system metadata; respectively constructing a data source node according to the data source metadata, constructing an ETL task node according to the ETL task metadata and constructing an index node according to the index system metadata in the graph database, and determining a node association relationship according to the data source metadata, the ETL task metadata and the index system metadata; constructing a data blood relationship chain taking the data source node as a starting point and the index node as an end point according to the node association relationship; and displaying the data blood relationship chain through the graph database. According to the method, a data warehouse table, an ETL scheduling task and an index system are integrated, and full-link blood relationship tracking and task monitoring operation and maintenance are achieved. The whole process of data from the data source to the index generation through the ETL task is visually presented, the workload of manually sorting the data relationship is greatly reduced, the data relationship can be changed along with the change of the data table structure, and the data blood relationship condition is flexibly and completely displayed.

The invention discloses an industrial data standard detection method based on a large model technology, and the method comprises the steps: 1, carrying out the definition of data standards, including a data content standard, a data format standard, a metadata standard, a data quality standard, a data exchange standard, a data security and privacy standard, and an industry and field specific standard; 2, performing large model knowledge enhancement, including general data standard integration, static standard rule scheme injection, dynamic semantic analysis detection scheme injection, data blood relationship detection scheme injection and composite detection mechanism fusion injection; and step 3, performing data standard detection, including data standard formatting and data standard automatic detection. According to the method disclosed by the invention, remarkable effect improvement is realized in the aspects of coverage rate detection, real-time performance, compliance interpretation capability, domain knowledge deep embedding, system management intelligence and the like, and a new generation of basic technical facilities is provided for industrial internet data management.

The invention discloses a data value evaluation method based on a data blood relationship, and relates to the technical field of data processing. The method comprises the following steps: constructing data blood relationship map data based on sensor metadata, communication protocol characteristics and a data blood relationship; obtaining a data value dynamic sorting result; constructing a monitoring index, importing the monitoring index into a tool to configure a data quality automatic monitoring tool, and obtaining a data exception intelligent detection result of the focused data node by using the data quality automatic monitoring tool; and performing data error repair and data processing flow optimization by using the block chain consensus and generative adversarial network to obtain a final data value evaluation result. According to the method, the data blood relationship graph can be efficiently constructed, the data value can be evaluated, the data quality can be monitored, and the data management strategy can be optimized, so that more accurate and reliable data support can be provided for decision making in intelligent traffic.

The invention relates to the technical field of network security, and discloses a data desensitization and access control method based on data consanguinity and sensitivity identification, which comprises the following steps of: constructing a data consanguinity graph for describing derivation, circulation and dependency relationships among fields; identifying a sensitivity level of each field based on a field consanguinity context provided by the graph; in response to a data access request of a user, dynamically generating an access control strategy for the target field in combination with a service scene, a user role and the sensitivity level of the target field, the strategy specifying access permission, access denial or access after desensitization; and executing the strategy through a data interceptor deployed in a data access key layer, releasing and shielding the data or outputting the data after real-time desensitization by calling a specified algorithm. According to the invention, accurate identification and dynamic and consistent fine-grained access control of the sensitivity of the network data are realized, and the security protection capability of the network data is improved.

The invention discloses a data processing system and method based on data consanguinity analysis, and relates to the technical field of big data analys.The data processing method comprises the steps that data full life cycle operation information is extracted, and a unique identifier containing a data source type code, an initial collection node ID and a global unique serial number is marked; preprocessing the operation information, and constructing dynamic association; constructing a data blood relationship network containing nodes and directional edges through dynamic association; judging whether the blood relationship is fractured according to the node number average value; and if so, repairing and generating an abnormal report. The system comprises an information extraction identification module, a preprocessing association module, a blood relationship network construction module, a blood relationship fracture judgment module and a fracture repair report module. According to the method, the problems of missing and breakage caused by data blood relationship static construction in the prior art are solved, the continuity, accuracy and traceability of data processing are improved, and the method is suitable for data quality control and problem traceability scenes.

The invention relates to the technical field of data analysis, in particular to an inter-application interface change control method and system based on data consanguinity analysis, and the method comprises the steps: 1, collecting data consanguinity; 2, constructing a weighted blood relationship map; step 3, change influence analysis; and 4, carrying out change evaluation. Compared with the prior art, the method and the system have the advantages that by constructing the weighted data consanguinity map, an upstream data source and a downstream dependency system of interface parameters between applications can be quickly positioned, and the interface change influence can be accurately predicted.

The invention discloses a multi-source heterogeneous big data aggregation analysis method based on master data modeling. The method comprises the following steps: modeling main data, defining a relation mode of a business entity and a data entity, and outputting a graph structure storage mode; classifying the service entities, and defining different data entity fusion algorithms for different data entity types by taking the main attributes of the service entities as the identification basis of the data entities; accessing data, and cleaning the accessed data; service entities and data entities are analyzed from the cleaned data based on a data entity fusion algorithm, and a relation theme library is constructed according to the incidence relation between the service entities and the data entities; storing the cleaned data and a relation theme library thereof; and carrying out modeling analysis on the cleaned data by combining the characteristics of the business entity and the relation theme library. According to the method, unified management of multi-source heterogeneous mass data is carried out from a global perspective, the blood relationship of the data is clear, and the authority is controllable.

The invention discloses an enterprise and government-oriented brain service integrated management system and method, and relates to the field of intelligent management, and the method comprises the steps: obtaining an industrial index abnormal event containing a multi-dimensional feature in real time, positioning a focus assembly in a service topology and upstream and downstream associated nodes thereof, and combining data blood relationship information to extract a key context; performing semantic embedding coding on an original log, and converting an unstructured log text into computable vector space representation; and finally, through a feature manifold search engine driven by a large model, cross-modal association matching is performed on the structured abnormal event features and the log semantic vectors, and an intelligent report is automatically generated. The system effectively solves the problems that in a traditional scheme, service link fault analysis and heterogeneous data association are difficult, and the dependence degree of artificial experience is too high, and the handling efficiency and decision-making reliability of abnormal events of a complex system are greatly improved.

The invention provides a data consanguinity establishing method, which comprises the following steps of: reading metadata of a technical index configured for a data product, and acquiring an identifier of the technical index from the metadata of the technical index; reading metadata of a business index configured for the data product, and obtaining an identifier of the business index and an identifier of a technical index associated with the business index from the metadata of the business index; reading configuration information of an application program of the data product, and obtaining an identifier of an application index related to the application program of the data product and an identifier of a technical index associated with the application index from the configuration information; and determining an association relationship among the technical index, the business index and the application index based on the identifier of the technical index, the identifier of the business index, the identifier of the technical index associated with the business index, the identifier of the application index and the identifier of the technical index associated with the application index. The invention further provides a data blood relationship establishing device, electronic equipment, a storage medium and a program product.

In some embodiments, the present disclosure provides an exemplary technically improved computer-based method utilizing an element level mapping module that determines a correlative relationships of interconnected nodes and edges in relation to an output data element and an input data element, and determines an appropriate graph of the relationships.

The embodiment of the invention provides a storage strategy optimization method and device based on data popularity and data consanguinity, and dynamic optimization of a storage strategy is realized through innovatively designing a multi-dimensional popularity calculation model and through node attribute analysis and a graph attention network. A directed graph structure is constructed, and a reliable dependency relationship management system is established in combination with blood relationship analysis. A hierarchical storage strategy is introduced, and the storage efficiency is ensured through heat division and dependency relationship optimization. According to the method, the defects of the traditional technology in the aspects of popularity evaluation, blood relationship analysis, storage optimization and the like are effectively overcome, and a technical guarantee is provided for software system storage management.

A system for a compliance-based data ecosystem that includes: a) a compliance management engine configured to enforce compliance rules and dynamically update policies based on industry regulations; b) a data governance module to classify, track and manage data lineage while enforcing retention and access policies; (c) a secure data processing unit that encrypts, anonymises and processes sensitive data while maintaining data protection and security; (d) an access control mechanism that implements role-based access control (RBAC), multi-factor authentication (MFA), and dynamic access assessments; (e) an audit and monitoring module that continuously and tamper-proof logs data transactions, access attempts and compliance-related actions; f) an AI-driven analytics engine that analyses data access patterns, detects anomalies and predicts potential compliance risks; and g) a blockchain-based validation system configured to record compliance actions and enforce compliance with smart contract-based policies, h) the system ensures secure, automated and compliant data management across multiple entities while complying with regulatory requirements.

The invention discloses a data consanguinity analysis method of an e-commerce data warehouse, a product, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: carrying out the entity analysis of a data consanguinity analysis question statement through a pre-training language model, and carrying out the vectorization processing of the obtained entity information, the method comprises the following steps: firstly, vectorizing an entity, then matching the vectorized entity with a vectorized entity in a vector database in a pre-created consanguinity knowledge graph, then querying an entity associated with the matched target vectorized entity from a graph database in the consanguinity knowledge graph, and finally inputting the entities matched from the two databases into a pre-trained language model. And data blood relationship analysis is carried out. Based on the pre-training language model and in combination with the blood relationship knowledge graph created based on the metadata management platform, automatic data blood relationship analysis is realized, the efficiency and the accuracy of data blood relationship analysis of the e-commerce data warehouse are improved, and the labor cost and the time cost are also saved.

The invention discloses a blood relationship analysis method, device and equipment for energy data and a storage medium, and the method comprises the steps: collecting station data corresponding to at least two station data sources in an energy station, and generating a data processing task corresponding to the station data; based on the data dependency relationship of the station data, executing the data processing task, and mapping the station data to a target map database; and traversing the target graph database on the basis of the exception type to which the exception data node in the target graph database belongs, determining an exception propagation path of the exception data node, and performing blood relationship analysis on the exception data node on the basis of the exception propagation path. According to the technical scheme, the data exception repairing efficiency is improved, and meanwhile the exception analysis accuracy of the exceptional data nodes is improved.

The invention provides a data management method and device, equipment and a storage medium, and is applied to the technical field of big data. The method comprises the steps of obtaining at least one data feature of first data; performing value score estimation according to the at least one data feature to obtain at least one first value score corresponding to the first data; under the condition that the first value score is not lower than a preset value, parent data or child data corresponding to the first data are obtained according to the data blood relationship corresponding to the first data, and second data are obtained; and updating the value score corresponding to the second data according to the first value score to obtain at least one second value score of the second data. According to the method, the importance of the data assets can be automatically identified, and the accuracy of value score evaluation of the data assets can be improved.