61 results about "Data lineage" patented technology

Tracking lineage of data. A method may be practiced in a network computing environment including a plurality of interconnected systems where data is shared between the systems. A method includes accessing a dataset. The dataset is associated with lineage metadata. The lineage metadata includes data indicating the original source of the data, one or more intermediary entities that have performed operations on the dataset, and the nature of operations performed on the dataset. A first entity performs an operation on the dataset. As a result of performing a first operation on the dataset, the method includes updating the lineage metadata to indicate that the first entity performed the operation on the dataset. The method further includes providing functionality for determining if the lineage metadata has been compromised in that the lineage metadata has been at least one of removed from association with the dataset, is corrupted, or is incomplete.

Presenting data lineage information by assigning a score to a data asset along a path between a data source and a data destination, where a predefined scoring function is applied to a characteristic of the data asset, and presenting via a computer-controlled output medium a description of the data source, the data destination, and the path between the data source and the data destination, where the description includes the data asset if the score meets predefined inclusion criteria.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

Provided are a techniques for viewing data lineage of objects. A data lineage view that includes at least one data lineage path is displayed. The data lineage view is generated by a data lineage tool that tracks an original object through processes that touched that original object. The at least one data lineage path is generated from the original object to a selected object and indicates how the original object was affected by the processes. The data lineage view is displayed as a fish eye view.

A system for providing data lineage information for data warehouse objects, the system including a plurality of job descriptions, a log for recording operational information of any of the jobs when any of the jobs are run, a plurality of schemas of databases accessed by the jobs, and a binding service configured to combine information from the job descriptions, the log, and the schemas to provide a data lineage for a data object of a data warehouse.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

The invention relates to a data linage analysis method and device. The method comprises analyzing query sentences based on mode configuration to recognize target tables, target fields, source tables and source fields in the query sentences; obtaining metadata defined by various database systems or users and performing accurate matching on fuzzy fields of the query sentences through the metadata; generating data lineage relationships of the query sentences according to the field tracing sequence of the recognized target fields and source fields; analyzing the data lineage relationships of a plurality of the query sentences through multi-layer sentence analysis. By means of the method and the device, data lineage of various general structured sentences can be analyzed flexibly.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

The invention relates to a data processing task relation setting method and system. The method includes the steps of obtaining at least one SQL script in a data processing task, carrying out morphology analysis and semantic analysis on SQL sentences in each SQL script in the at least one SQL script to build a data lineage relation of the SQL sentences, building a data lineage relation of the SQL scripts according to the data lineage relation of the SQL sentences, building a data lineage relation of the data processing task according to the data lineage relation of the SQL scripts in the at least one SQL script, determining data input and output of a data level and a task level of the data processing task, and determining and setting the relation between the data processing task and another data processing task according to the data lineage relation and the data level of the data processing task. Intelligent analysis and setting of the relation of the SQL data processing tasks can be achieved, the automation degree of data task scheduling configuration is improved and accuracy and efficiency of data operation and maintenance are achieved.

Techniques are disclosed for capturing and visualizing data lineage in content management systems. For example, a method comprises the following steps. A plurality of data sets is received. Each of the data sets is associated with a party and comprises a plurality of information. A set of lineage data about one or more of the data sets is received. The lineage data comprises information about the history of a particular data set. A user interface is presented that conveys a representation of one or more of the plurality of received data sets and at least a portion of the lineage data about the history of one or more of the data sets. A command is received at the user interface to merge or unmerge two data sets in the plurality of data sets. Two or more data sets in the plurality of data sets are merged or unmerged based on the received command.

In an example, one or more processes that transform data originating in one or more of a plurality of different data sources are identified. Then, for each of the identified one or more processes, a relationship table is created, the relationship table listing objects input to the process, objects output from the process, and one or more mappings defining transformations performed on the objects input to the process to produce the objects output from the process. Dependencies are created between relationship tables by linking at least an instance of a first object input to a process in a first relationship table with at least an instance of the first object output from a process in a second relationship table. Lineage of an object in the computer system is traced by accessing one or more of the relationship tables and the dependencies.

The invention discloses a sql automatic semantic analysis-based data lineage analysis system. The system comprises a sql preprocessing module, a lineage recognition module and a lineage display module connected in sequence, wherein the sql preprocessing module is used for establishing a keyword rule library, reading a to-be-detected data model structure and a data processing sql script from a database where to-be-detected data is located, and decomposing the data processing sql script to form a script analysis table; the lineage recognition module is used for carrying recognizing a keyword of the data processing sql script read from the sql preprocessing module, extracting lineage information from the data processing sql script corresponding to the keyword and storing the lineage information into the script analysis table. The invention furthermore discloses an analysis method of the sql automatic semantic analysis-based data lineage analysis system. According to the system and the method, lineage analysis can be carried out on other data on the basis of an ETL processing process.

An example system includes a processor that can generate a first lineage graph based on a first set of monitored assets and processes used to produce a data asset. The processor can detect a data quality issue at the data asset. The processor can also generate a second lineage graph including a second set of monitored assets and processes that produced the data asset with the data quality issue. The processor can further compare the second lineage graph with the first lineage graph to detect a potential root cause of the data quality issue. The processor can also further modify an asset or process corresponding to the potential root cause of the data quality issue.

A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

The invention relates to an ETL data lineage query system and query method. The ETL data lineage query system is characterized by comprising an operation module and a data lineage management module, the operation module is capable of running task scripts, dividing tasks, generating a task script file containing operation information and transmitting a task division file containing the operation information to the data lineage management module; the data lineage management module is capable of receiving a user configuration file, collecting a source data file, the task script file containing the operation information and storing data lineage information.

A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

A system for providing data lineage information for data warehouse objects, the system including a plurality of job descriptions, a log for recording operational information of any of the jobs when any of the jobs are run, a plurality of schemas of databases accessed by the jobs, and a binding service configured to combine information from the job descriptions, the log, and the schemas to provide a data lineage for a data object of a data warehouse.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

The invention belongs to the technical field of data lineage, and particularly relates to an inspection business collaborative process-oriented inspection business lineage data acquisition and integration method. For an inspection business scene, the method comprises the following steps: designing an inspection business process data acquisition method for collecting inspection business process execution process data; and designing an inspection business process lineage data integration method, and converting the collected process execution process data into process lineage data. The method provided by the invention is packaged into a flow system server in a service form. According to the invention, the requirements of inspection business process data collection and lineage integration in an inspection business scene can be met.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for metadata management. One of the methods includes receiving user input selecting a first node. The method includes receiving a first data lineage of a first object, the first object having a type, the first data lineage describing relationships between the first object and one or more datasets or transforms. The method includes receiving user input selecting a second node. The method includes receiving a second data lineage of a second object, the second object having the same type as the first object. The method includes performing a comparison of the first node and the first data lineage to the second node and the second data lineage. The method includes generating a report based on the comparison.

Embodiments disclosed herein provide systems, methods, and computer readable media for data lineage based multi-data store recovery. In a particular embodiment, a method provides identifying first data in a first table of a plurality of tables stored in a plurality of data stores and restoring the first data to a first correct version of the first data in a prior version of the first table. The method further provides identifying a second table of the plurality of tables that descends from the first table and includes second descendent data that stems from the first data. The method also provides restoring the second descendent data to a second correct version of the second descendent data in a prior version of the second table.

A method and system for managing and tracking the pedigree or data lineage of an electronic document. The methods and systems provide a standardized way for managing the pedigree of an electronic document regardless of its data type.

Techniques are disclosed for capturing and visualizing data lineage in content management systems. For example, a method comprises the following steps. A plurality of data sets is received. Each of the data sets is associated with a party and comprises a plurality of information. A set of lineage data about one or more of the data sets is received. The lineage data comprises information about the history of a particular data set. A user interface is presented that conveys a representation of one or more of the plurality of received data sets and at least a portion of the lineage data about the history of one or more of the data sets. A command is received at the user interface to merge or unmerge two data sets in the plurality of data sets. Two or more data sets in the plurality of data sets are merged or unmerged based on the received command.

An example system includes a processor that can generate a first lineage graph based on a first set of monitored assets and processes used to produce a data asset. The processor can detect a data quality issue at the data asset. The processor can also generate a second lineage graph including a second set of monitored assets and processes that produced the data asset with the data quality issue. The processor can further compare the second lineage graph with the first lineage graph to detect a potential root cause of the data quality issue. The processor can also further modify an asset or process corresponding to the potential root cause of the data quality issue.

A method includes receiving information indicative of an output dataset generated by a data processing system; identifying, based on data lineage information relating to the output dataset, one or more upstream datasets on which the output dataset depends; analyzing one or more of the identified one or more upstream datasets on which the output dataset depends. The analyzing includes, for each particular upstream dataset of the one or more upstream datasets, applying one or more of: (i) a first rule indicative of an allowable deviation between a profile of the particular upstream dataset and areference profile for the particular upstream dataset, and (ii) a second rule indicative of one or more allowable values or prohibited values for each of one or more data elements in the particular upstream dataset, and based on the results of applying the one or more rules, selecting one or more of the upstream datasets. The method includes outputting information associated with the selected oneor more upstream datasets.

Tracking lineage of data. A method may be practiced in a network computing environment including a plurality of interconnected systems where data is shared between the systems. A method includes accessing a dataset. The dataset is associated with lineage metadata. The lineage metadata includes data indicating the original source of the data, one or more intermediary entities that have performed operations on the dataset, and the nature of operations performed on the dataset. A first entity performs an operation on the dataset. As a result of performing a first operation on the dataset, the method includes updating the lineage metadata to indicate that the first entity performed the operation on the dataset. The method further includes providing functionality for determining if the lineage metadata has been compromised in that the lineage metadata has been at least one of removed from association with the dataset, is corrupted, or is incomplete.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.