81 results about "Data lineage" patented technology

The invention relates to the technical field of network security, and discloses a data desensitization and access control method based on data consanguinity and sensitivity identification, which comprises the following steps of: constructing a data consanguinity graph for describing derivation, circulation and dependency relationships among fields; identifying a sensitivity level of each field based on a field consanguinity context provided by the graph; in response to a data access request of a user, dynamically generating an access control strategy for the target field in combination with a service scene, a user role and the sensitivity level of the target field, the strategy specifying access permission, access denial or access after desensitization; and executing the strategy through a data interceptor deployed in a data access key layer, releasing and shielding the data or outputting the data after real-time desensitization by calling a specified algorithm. According to the invention, accurate identification and dynamic and consistent fine-grained access control of the sensitivity of the network data are realized, and the security protection capability of the network data is improved.

The invention relates to the technical field of urban agent decision-making auditing, and discloses an urban agent decision-making auditing method and system based on data consanguinity tracking, and the method comprises the steps: collecting metadata of a whole urban agent decision-making process in a layered manner, integrating multiple layers of metadata, and carrying out the multi-dimensional marking and dependence recognition; constructing a time sequence blood relationship map, and performing storage optimization; performing decision contribution degree evaluation on the target decision node; restoring a key path of the target decision node, and analyzing a causal association relationship; performing multi-level auditing analysis on the decision-making process, and identifying a root cause node; generating a decision audit report; analyzing historical influence and future risk execution of the root cause problem; auditing information is extracted and integrated; matching a treatment scheme and pushing and executing the treatment scheme; tracking the treatment effect and continuously optimizing; according to the method, the blood relationship tracking system covering the whole process from data acquisition to decision output is constructed, so that the transparency and auditing of the urban agent decision process are realized.

The invention relates to the technical field of data processing and artificial intelligence application, and discloses a data quality rule automatic generation and self-adaption method, system and device and a medium. Performing business term inference and semantic modeling on the structured metadata to construct business concepts and relationships; analyzing the processing logic of the structured metadata based on the business concept and the relationship to identify a data consanguinity map; building a power grid domain knowledge graph based on a business concept, a semantic model and a consanguinity graph mining hidden relationship; reasoning based on the power grid domain knowledge graph and historical data to generate data quality rules and business constraints; monitoring the execution effect of the data quality rule, and performing adaptive optimization through quantitative evaluation; and executing the self-adaptively optimized rule, marking abnormal data and feeding back an execution result. According to the method, the automation and intelligence level of quality treatment can be improved, and continuous evolution and vitality of a data quality system are ensured.

The invention provides a data consanguinity analysis method, device and equipment, a medium and a product, and relates to the technical field of data processing.The method comprises the steps that an initial consanguinity graph of data in a data lake is established; according to the structure of the data table of the data source end and the target data, establishing a full-link data blood relationship corresponding to the data table of the data source end; wherein the target data comprises test data and / or feature data; and correcting the initial blood relationship map according to the full-link data blood relationship to obtain a target blood relationship map. According to the scheme, the initial blood relationship is corrected by using the full-link data blood relationship, cross-platform data flow tracking can be realized, and the accuracy of data blood relationship identification is improved.

This invention relates to the field of data processing and discloses an intelligent cleaning method for multi-source heterogeneous big data. The method includes: collecting structured, semi-structured, and unstructured data from multiple source systems and parsing them to obtain raw data units; performing standard concept mapping on the raw data units and writing source identifiers, extraction method identifiers, collection time identifiers, upstream task identifiers, and derived link identifiers to generate data units with lineage identifiers; forming candidate association clusters based on stability identifiers, contextual features, and temporal adjacency relationships; constructing a conflict evidence graph within the candidate association clusters and determining edge weights by combining source credibility, time freshness, lineage propagation depth, and source drift coefficient; locating conflict-connected subgraphs and determining the minimum repair boundary by combining the data lineage graph; generating multiple candidate repair strategies and arbitrating them.

A system and a method for data protection assessment using threat modelling and adaptive optimization is disclosed. The system (100) comprising a processor (105) and memory (110) with instructions to receive a data access request (345) from entities (120) via authenticated digital interface (125) including structured and unstructured data categories (350), metadata parameters (355), initiating contextual analysis to establish baseline compliance parameters (360). The system analyses data category and metadata using adaptive classification logic and correlation metrics across stored data lineage, distinguishing privacy -critical from non- sensitive data. Threat modelling (130) evaluates correlations among privacy attributes (365), metadata, operational parameters, and regulatory requirements to identify vulnerabilities (375) and exposure points. A composite risk index (135) is derived, forming a structured risk profile (140) with quantified scores (380). Compliance recommendations (150) are generated, safeguard parameters (155) are derived, authorization validated by a compliance authority (160), and a compliance trace (388) is recorded.

A data platform for providing large scale data discovery with near real time data cataloguing and detailed lineage. Data is received from one or more sources, and the received data is written to a data storage systems. The received data is processed at the data storage systems using sensors at the data storage systems. Metadata and data definitions associated with the data are sent to a data service in near real-time. A data catalog is generated at the data service based on the metadata and data definitions associated with the data. Data lineage for the data is created at the data service. The data catalog and data lineage are presented at a data discovery interface.

The application discloses a data blood relationship mining method and device of an advanced enterprise application programming language, and the data blood relationship mining method comprises the following steps: performing word segmentation processing on the code of the advanced enterprise application programming language to obtain a mark list; wherein the word segmentation processing is performed in the minimum unit of words and symbols; the mark list is parsed according to the grammar rules of the Backus Normal Form to obtain a semantic expression list, and the semantic expression list comprises grammar keywords and grammar contents; the semantic expression list is formatted according to the grammar rules of the Backus Normal Form to obtain a semantic expression dictionary list; the semantic expression dictionary list is parsed, nodes are created, the blood relationship between different nodes is found and saved. The ABAP program code is subjected to fine-grained word segmentation processing, and semantic analysis is performed on the basis of the fine-grained word segmentation processing, so that the association relationship between data is finally obtained, and the blood relationship between nodes is determined.

The present disclosure provides a data lineage generation method and device, electronic equipment and storage medium, relates to the technical field of computers, in particular to the technical field of data mining, data production processing and information flow. The specific implementation scheme is: in response to receiving a data lineage generation request, determining a target data node of a data lineage to be generated in a target directed acyclic graph; determining a target path in the target directed acyclic graph; obtaining the assignment data of the specified data node in the target path; according to the assignment data, executing a data generation process according to the target path to obtain target data; and generating a data lineage about the target data node according to the target data, the assignment data and the target path. Through the present scheme, a more fine-grained data lineage can be generated.

An information handling system operating an integration connectivity sales facilitation system may comprise a display device displaying a request from a purchasing user to facilitate purchase of a product for sale by the selling user, a processor to automatically generate code instructions of a future data integration process for transferring a first dataset field value identified by a first dataset field name at a purchasing user to a second dataset field value identified by a second dataset field name at the selling user, based on a merged data lineage map for previously generated code instructions for transferring such first and second dataset field values to a common trading partner storage location identified by a common trading partner dataset field name, and a network interface device transmitting the code instructions for future execution and a runtime engine for execution at a remote location in deployment of the future data integration process.

This disclosure provides a method and system for constructing and inferring data lineage for farmland quality monitoring, applicable to the fields of farmland quality monitoring and geographic information data governance. The method includes acquiring multi-source heterogeneous farmland data from a target area, organizing and standardizing it to generate standard farmland quality monitoring indicator data; then, based on the standard farmland quality monitoring indicator data, constructing a farmland quality evaluation system, generating evaluation result data and lineage metadata; further, based on the evaluation result data and lineage metadata, constructing a farmland quality lineage knowledge graph; and visualizing the farmland quality lineage knowledge graph to enable visualized traceability and inference of farmland quality monitoring data lineage. In this way, a fully traceable lineage network can be constructed, and based on this network, visualized queries and bidirectional inference can be achieved, thereby realizing full-link transparency and computable dynamic impact of the evaluation process.

The invention provides a data blood relationship generation method and device, electronic equipment and a storage medium. Relates to the technical field of data traceability. The fields in the script data are positioned through the keyword template, vector conversion of meta-characters, capital and lowercase and dynamic parameters is realized in combination with a vector library vectorization conversion rule, and the accuracy and suitability of field conversion are improved. Field vectorization, rule matching conversion and mapping relation establishment are completed through the conversion model, the conversion data and the blood relationship are directly output, the generation process of the blood relationship is simplified, and automation and high efficiency of data blood relationship establishment are improved. The blood relationship is defined according to the mapping relationship between the script data and the conversion data, the association before and after data conversion is clearly traced, and the accuracy and traceability of the blood relationship are guaranteed. The conversion rules stored in the vector library can provide reusable rule support for subsequent similar script processing, the expansibility and reusability of the scheme are improved, and the quality and efficiency of cross-database script data blood relationship generation are integrally guaranteed.

This application discloses a self-attention recommendation method based on data lineage features and behavioral semantic features, comprising: training a product recommendation model based on a neural network using users' historical interaction sequence information and data lineage relationship information involving products; the product recommendation model including a data lineage feature extraction module, a behavioral semantic modeling module, and a self-attention decoder; acquiring the target user's historical interaction sequence information and the data lineage relationship information involving products and inputting them into the trained product recommendation model, and outputting the corresponding target recommended product. This invention also provides a self-attention recommendation device and apparatus based on data lineage features and behavioral semantic features. The product recommendation model trained by this method can further utilize data lineage features and user behavioral semantic features, thus enabling more accurate item recommendations.

This invention relates to data analysis technology and discloses a data lineage analysis method, comprising: acquiring raw data from real-time and offline data streams; identifying the data type of the raw data and selecting a preset parsing model based on the data type; parsing the raw data using the preset parsing model to obtain the data nodes, task nodes, and data flow relationships between each data node and task node; and constructing a lineage graph based on the data nodes, task nodes, and data flow relationships. Furthermore, this invention also relates to blockchain technology, allowing test reports to be stored on blockchain nodes. This invention also proposes a data lineage analysis method device, electronic device, and storage medium. This invention can improve the accuracy of lineage relationships for electronic transaction data.

Various embodiments are generally directed to techniques for building data lineages for queries, such as SQL queries. Some embodiments are particularly directed to a lineage tool that is able to construct data lineages in a recursive manner that uses the text of a query to identify dependent tables. In several embodiments, the data lineage tool may parse SQL queries to identify columns and dependent tables, including analyzing interdependent queries used to populate dependent tables and proceeding until the true source of data is identified. In several embodiments, the data lineage tool may utilize the relationships and dependencies to build element and table level lineages.

This invention relates to the field of data governance technology, specifically to an intelligent inventory system for medical data resources. The system comprises: first, performing connection testing, collection scope detection, and access permission assessment on the target data source, and collecting medical metadata to form a metadata set; then, performing semantic association analysis and data lineage analysis on the metadata set, and generating classification results and corresponding evidence chains by combining data classification standards, data classification rules, and an intelligent recognition model; finally, outputting the data resource inventory results, and performing verification and feedback updates when inconsistencies are identified or when manual confirmation instructions are received. This invention can improve the completeness of metadata collection, classification accuracy, and result traceability.

Blockchain-verified data tracking enables comprehensive and secure management of personal data across distributed systems. Various implementations involve recording metadata about data movements, access events, and modifications in a decentralized ledger, allowing for real-time tracking and verification of data lineage. Multiple blockchain ledgers may be implemented, with varying levels of detail and access permissions, to balance transparency with data sensitivity.

Techniques are disclosed relating to a graphical user interface (GUI) that is operable to depict data lineage information in levels. In some embodiments, data lineage information may specify a directed graph that is indicative of a data lineage associated with a plurality of data elements. For example, in the data lineage information, the plurality of data elements may be represented as a corresponding plurality of nodes and, in the directed graph, the plurality of nodes may be connected by edges in a manner that is indicative of the data lineage relationships between the plurality of data elements. In various embodiments, the disclosed techniques may generate a data lineage GUI that, for a selected data element of the plurality of elements, is usable to navigate different levels of the data lineage in an upstream and downstream direction relative to a particular level of the selected data element.

The application discloses a data blood relationship analysis method and device and a computer readable storage medium. The method comprises the following steps: obtaining job blood relationship data based on a directed graph corresponding to a scheduling task in a scheduling system, and obtaining a first application program identifier based on an execution log corresponding to the scheduling task; when a structured query language (SQL) statement is detected, obtaining field blood relationship data corresponding to the SQL statement and a second application program identifier corresponding to the field blood relationship data; determining a job blood relationship subgraph based on the job blood relationship data and a Neo4j point-edge model, and determining a table field blood relationship subgraph based on the Neo4j point-edge model and the field blood relationship data; and connecting the job blood relationship subgraph and the table field blood relationship subgraph based on the first application program identifier and the second application program identifier. The application can obtain blood relationship data of different types of data systems, adapt the data blood relationship analysis method to different types of data systems, and improve the expansibility of the data blood relationship analysis method.

In some implementations, a system may receive information identifying a dataset. The system may process an identification attribute using a function that generates a first value, to generate a first identifier for the dataset. The system may search a data store storing a plurality of groupings to identify a grouping with the first identifier for the dataset. The system may extract a second identifier from the grouping with the first identifier for the dataset. The system may search a data lineage based graph representation of a plurality of datasets to identify a graph node representing the dataset. The system may update the data lineage based graph representation of the plurality of datasets to link the dataset with the at least one other dataset based on searching the data lineage based graph representation of the plurality of datasets to identify the graph node.

The invention relates to the technical field of data processing, and particularly provides a real-time data warehouse incremental updating method, system and device and a storage medium, and the method comprises the steps: obtaining an identifier of a target data table in response to a captured data change event of the target data table in a source business database; based on the identifier of the target data table, querying a pre-constructed consanguinity knowledge base to determine one or more associated data tables influenced by the data change event; according to a dependency relationship in the blood relationship knowledge base, sorting the one or more associated data tables to generate an incremental update task chain; and performing incremental data updating on the one or more associated data tables according to the incremental updating task chain. According to the invention, by introducing the intelligent data blood relationship map, the precision and automation of the incremental updating process are realized.

This invention discloses a real-time data synchronization method based on DAG orchestration and asynchronous zero-copy architecture, comprising: constructing a directed acyclic graph (DAG) based on timestamped data and its data lineage to obtain parallelizable task nodes; sending data using a zero-copy transmission method based on io_uring based on the parallelizable task nodes to obtain zero-copy transmitted data; grouping the received data according to the source IP address and storing it in the corresponding receiving queue to obtain grouped queue data; and sorting and merging the grouped queue data according to the global timestamp using a time window multi-path merge algorithm, and using an event barrier mechanism to handle the order of data definition language instructions and data manipulation language instructions to obtain a globally ordered event stream. This invention achieves dependency-aware parallel scheduling, zero-copy asynchronous transmission, and globally ordered merging of cross-source events.

This application discloses a visualization method, system, and medium for bidirectional linking of data resource catalogs and storage. The method includes: obtaining the logical data catalog and physical storage structure, where the catalog contains logical objects and the storage contains physical objects; receiving the user-selected mapping mode and configuring corresponding mapping rules; constructing a bidirectional reversible mapping relationship between logical and physical objects based on the rules, including forward and reverse mapping functions; persistently storing the mapping relationship in a mapping relationship database and generating version information. This application overcomes the drawbacks of independent catalogs and storage, allowing business personnel to locate physical storage locations from the catalog and technical personnel to locate business items from fields. Version management enables change recording, retrospection, and accountability, laying a technical foundation for data lineage and collaborative governance.

A network security monitoring system having a plurality of data sources for providing security data, an extraction unit for extracting the security data from the sources of the security data to form extracted security data, a graphing unit for mapping the extracted security data to at least one or more data fields and data models to form graph data, and a data lineage determination unit for determining a data lineage between the log data and one or more of the alerts based on the graph data.

Computer-implemented methods and systems for improving the efficiency of creating source-to-target mapping documentation are provided. Some embodiments involve receiving a template for creating a code script that executes data transformations. Some receiving the code script for executing data transformations. Some embodiments involve comparing the structure of a code script against the template and providing a warning for a portion of the code script that fails to conform to the template based on the comparing. Some embodiments involve providing the code script and the data transformation results to a documentation generator. Some embodiments involve analyzing the code script and the data transformation results. In some embodiments, the analysis includes identifying a source and target as executed by the code script and the data transformations. Some embodiments involve generating an output document that shows the data lineage of an identified source data.

A data blood relationship determination method, device, equipment, storage medium and program product are disclosed. The present application relates to the technical field of databases. The method comprises: obtaining a data processing log, a job running record and a set mapping relationship; analyzing the data processing log to obtain a plurality of initial database operation statements; filtering the plurality of initial database operation statements based on the job running record and the set mapping relationship to obtain at least one target database operation statement; and determining a data blood relationship based on the at least one target database operation statement; wherein the data blood relationship comprises a table blood relationship and a field blood relationship. According to the scheme of the present application, the database operation statements are filtered based on the job running record and the set mapping relationship, and the data blood relationship is determined based on the filtered database operation statements, which can improve the efficiency and accuracy of data blood relationship determination.

This application discloses a data processing method, apparatus, device, medium, and product, belonging to the blockchain field. The method includes: acquiring multiple secret shares sent by a data provider, wherein the secret shares are obtained by the data provider from splitting the lineage of a de-identified local data product, wherein the data provider includes at least one data provider, and the secret shares sent by different data providers have the same location index; and locally aggregating the secret shares with the same location index from at least one data provider to obtain a global data lineage. In this invention, each institution transmits secret shares to a secure computing node with a trusted execution environment through a secure channel, and aggregating the same portion of secret shares transmitted by various institutions to form a merged global data lineage, ensuring that the data achieves traceability while not leaking privacy information, and ensuring the security of data transmission and aggregation processes.

This invention relates to a method and system for cross-regional data end-to-end traceability and reassembly in a power physical isolation environment. The method involves intercepting service data at the source end in Security Zone 1, constructing a collision-resistant composite content fingerprint using hash digests and payload length, and building an independent traceability packet containing both the fingerprint and TraceID. The service data and traceability packet are then unidirectionally transferred to Security Zone 3 via isolation devices. At the destination end, a double-buffered queue is established, and a time-sliding window algorithm is used, indexed by the content fingerprint, to re-associate and match the received out-of-order service data with the traceability packet, thereby restoring the entire data lineage. This invention effectively achieves end-to-end data auditing and fault determination across security zones without modifying the physical isolation devices or core business systems, and features non-intrusiveness, high robustness, and high security.

The invention discloses a law-oriented data set construction method and device and a medium, and the method comprises the steps: collecting original data of a plurality of law data sources, and carrying out the evidence chain standardization processing of the collected data, and generating standardized data; performing pre-labeling, manual labeling and expert verification on the standardized data through a preset legal domain model to generate final labeling data; performing quality control, credibility evaluation and data enhancement processing on the final annotation data to generate enhanced data; performing version management and data consanguinity tracing on the enhanced data to generate versioned data; and performing legal reasoning and evidence chain verification on the versioned data to generate an interpretable data set. Through unified collection of multi-source authoritative data and full-process automatic scheduling, the legal data set construction period is greatly shortened, the labeling efficiency is remarkably improved, manual intervention is reduced, and the efficiency bottleneck caused by traditional data collection dispersion and tedious process is solved.

This invention relates to the field of intelligent manufacturing technology, and particularly to an IoT-based system for coding and integrating home furnishing products into an ERP system. The system uses a production sensing module to verify the historical convergence of equipment operating parameters via Mahalanobis distance to obtain equipment status confidence. A coding decision module generates a target coding strategy based on the fusion characteristics of production data and an environmental disturbance index compensation model, and then filters it using a strategy effectiveness prediction model. An edge verification module achieves dynamic error correction and dynamic reconstruction of error-correcting codes through bidirectional verification of code graph semantic integrity and a conflict resolution decision matrix. An ERP synchronization module achieves causal consistency updates of inventory status through topological sorting of the coded data causal graph and transaction compensation. A quality traceability module constructs a product traceability knowledge graph to achieve entity disambiguation and data lineage traceability. This invention solves the technical problems of data silos and broken traceability chains in production coding and ERP systems.