Data Lineage Verification via Automated and Manual Association

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data processing systems face challenges in generating accurate and complete data lineage information, with manual methods being inaccurate and outdated, while automated methods produce complex, technical output that is difficult for users to understand and interpret.

Innovation Solution

A system that combines manually generated user-specified data lineage with automatically derived data lineage, creating an association between the two to verify accuracy, identify discrepancies, and map technical terminology to business-friendly terms, facilitating error identification and data quality assessment.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If automated methods are used to generate data lineage information, then productivity is improved, but the output becomes complex and difficult for users to understand

Engineering Contradiction:

Improvedata lineage generation efficiencyVSAvoiduser understanding of data lineage output

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent introduces an intermediary layer that translates automated technical data lineage output into user-friendly visual representations. This intermediary processing layer converts complex automated results into comprehensible formats without sacrificing the efficiency benefits of automation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system changes the presentation parameters of data lineage information based on user needs. It transforms the same underlying data into different formats (technical vs. business-friendly) by adjusting visualization parameters, making the information accessible to different user groups while maintaining automated generation efficiency.

Inventive Principle:

Principle #35Parameter changes

2Ease of operation

If manual methods are used to generate data lineage information, then ease of operation is improved, but accuracy and completeness deteriorate

Engineering Contradiction:

Improveuser control over data lineage generationVSAvoiddata lineage accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent merges manual user-specified data lineage information with automatically derived data lineage information. This combination allows users to provide high-level guidance while automated systems fill in detailed accuracy requirements, achieving both user control and measurement precision.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system implements feedback mechanisms where user-specified lineage information is compared against automatically derived information. Discrepancies are identified and resolved through iterative refinement, ensuring both user control and high accuracy in the final data lineage output.

Inventive Principle:

Principle #23Feedback

3Measurement precision

If user-specified data lineage is combined with automatically derived data lineage, then measurement precision is improved, but device complexity increases

Engineering Contradiction:

Improvedata lineage verification accuracyVSAvoidsystem architecture complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the data lineage generation process into distinct modules: user input processing, automated derivation, association matching, and discrepancy analysis. This segmentation manages complexity by organizing functions into independent, manageable components that can be developed and maintained separately.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system employs universal data structures and processing mechanisms that handle both user-specified and automatically derived lineage information through the same association and comparison logic. This multi-functionality reduces overall system complexity by reusing components across different input types.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11226957B2Systems and methods for determining relationships among data elements

Publication Date: 2022.01.18 AB INITIO TECHNOLOGY LLC

AI summary

A data processing system configured to perform: obtaining a first data lineage representing relationships among physical data elements, the first data lineage being generated at least in part by performing at least one of: (a) analyzing source code of at least one computer program configured to access the physical data elements; and (b) analyzing information obtained during runtime of the at least one computer program; obtaining, based on user input, a second data lineage representing relationships among business data elements; obtaining an association between at least some of the physical data elements of the first data lineage and at least some of the business data elements of the second data lineage; and generating, based on the association between the physical data elements and the business data elements, an indication of agreement or discrepancy between the first data lineage and the second data lineage.