60 results about "Data lineage" patented technology

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

A system for providing data lineage information for data warehouse objects, the system including a plurality of job descriptions, a log for recording operational information of any of the jobs when any of the jobs are run, a plurality of schemas of databases accessed by the jobs, and a binding service configured to combine information from the job descriptions, the log, and the schemas to provide a data lineage for a data object of a data warehouse.

The invention relates to a data linage analysis method and device. The method comprises analyzing query sentences based on mode configuration to recognize target tables, target fields, source tables and source fields in the query sentences; obtaining metadata defined by various database systems or users and performing accurate matching on fuzzy fields of the query sentences through the metadata; generating data lineage relationships of the query sentences according to the field tracing sequence of the recognized target fields and source fields; analyzing the data lineage relationships of a plurality of the query sentences through multi-layer sentence analysis. By means of the method and the device, data lineage of various general structured sentences can be analyzed flexibly.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

The invention relates to a data processing task relation setting method and system. The method includes the steps of obtaining at least one SQL script in a data processing task, carrying out morphology analysis and semantic analysis on SQL sentences in each SQL script in the at least one SQL script to build a data lineage relation of the SQL sentences, building a data lineage relation of the SQL scripts according to the data lineage relation of the SQL sentences, building a data lineage relation of the data processing task according to the data lineage relation of the SQL scripts in the at least one SQL script, determining data input and output of a data level and a task level of the data processing task, and determining and setting the relation between the data processing task and another data processing task according to the data lineage relation and the data level of the data processing task. Intelligent analysis and setting of the relation of the SQL data processing tasks can be achieved, the automation degree of data task scheduling configuration is improved and accuracy and efficiency of data operation and maintenance are achieved.

Techniques are disclosed for capturing and visualizing data lineage in content management systems. For example, a method comprises the following steps. A plurality of data sets is received. Each of the data sets is associated with a party and comprises a plurality of information. A set of lineage data about one or more of the data sets is received. The lineage data comprises information about the history of a particular data set. A user interface is presented that conveys a representation of one or more of the plurality of received data sets and at least a portion of the lineage data about the history of one or more of the data sets. A command is received at the user interface to merge or unmerge two data sets in the plurality of data sets. Two or more data sets in the plurality of data sets are merged or unmerged based on the received command.

The invention discloses a sql automatic semantic analysis-based data lineage analysis system. The system comprises a sql preprocessing module, a lineage recognition module and a lineage display module connected in sequence, wherein the sql preprocessing module is used for establishing a keyword rule library, reading a to-be-detected data model structure and a data processing sql script from a database where to-be-detected data is located, and decomposing the data processing sql script to form a script analysis table; the lineage recognition module is used for carrying recognizing a keyword of the data processing sql script read from the sql preprocessing module, extracting lineage information from the data processing sql script corresponding to the keyword and storing the lineage information into the script analysis table. The invention furthermore discloses an analysis method of the sql automatic semantic analysis-based data lineage analysis system. According to the system and the method, lineage analysis can be carried out on other data on the basis of an ETL processing process.

The invention relates to an ETL data lineage query system and query method. The ETL data lineage query system is characterized by comprising an operation module and a data lineage management module, the operation module is capable of running task scripts, dividing tasks, generating a task script file containing operation information and transmitting a task division file containing the operation information to the data lineage management module; the data lineage management module is capable of receiving a user configuration file, collecting a source data file, the task script file containing the operation information and storing data lineage information.

A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for metadata management. One of the methods includes receiving user input selecting a first node. The method includes receiving a first data lineage of a first object, the first object having a type, the first data lineage describing relationships between the first object and one or more datasets or transforms. The method includes receiving user input selecting a second node. The method includes receiving a second data lineage of a second object, the second object having the same type as the first object. The method includes performing a comparison of the first node and the first data lineage to the second node and the second data lineage. The method includes generating a report based on the comparison.

Embodiments disclosed herein provide systems, methods, and computer readable media for data lineage based multi-data store recovery. In a particular embodiment, a method provides identifying first data in a first table of a plurality of tables stored in a plurality of data stores and restoring the first data to a first correct version of the first data in a prior version of the first table. The method further provides identifying a second table of the plurality of tables that descends from the first table and includes second descendent data that stems from the first data. The method also provides restoring the second descendent data to a second correct version of the second descendent data in a prior version of the second table.

A method and system for managing and tracking the pedigree or data lineage of an electronic document. The methods and systems provide a standardized way for managing the pedigree of an electronic document regardless of its data type.

An example system includes a processor that can generate a first lineage graph based on a first set of monitored assets and processes used to produce a data asset. The processor can detect a data quality issue at the data asset. The processor can also generate a second lineage graph including a second set of monitored assets and processes that produced the data asset with the data quality issue. The processor can further compare the second lineage graph with the first lineage graph to detect a potential root cause of the data quality issue. The processor can also further modify an asset or process corresponding to the potential root cause of the data quality issue.

Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.