Data Lineage Error Tracking for Model Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern businesses face challenges in tracking errors in data set lineage due to complex queries and varying user perspectives on data from cloud-based data warehouses, leading to difficulties in maintaining data integrity and consistency across different user analyses.
Innovation Solution
A system and method for tracking errors in data set lineage by receiving changes to a model data set, accessing dependent worksheets, rebuilding them with the changes, and generating error reports to identify and present errors to users, ensuring data consistency and integrity across dependent analyses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complex queries are constructed to retrieve and organize data from cloud-based data warehouses in different ways, then multiple different views of the same data can be created, but data integrity and consistency become difficult to maintain across different user analyses
Solution Approach 1:
The system performs preliminary actions by automatically detecting changes to model data sets before they propagate to dependent worksheets. It proactively identifies affected worksheets, rebuilds them with the changes, and generates error reports in advance, preventing data integrity issues rather than reacting to them after occurrence.
Solution Approach 2:
The system implements feedback by automatically tracking and reporting errors in dependent worksheets when model data sets change. The error report provides feedback to users about which worksheets are affected by changes, enabling them to maintain data integrity while using flexible data views.
2Reliability
If automatic error tracking and reporting for dependent worksheets is implemented, then data integrity and consistency are maintained, but system complexity increases
Solution Approach 1:
The system performs self-service by automatically detecting changes to model data sets, identifying affected dependent worksheets, rebuilding them, and generating error reports without requiring manual intervention. This automation maintains data consistency while minimizing the operational complexity for users.
Solution Approach 2:
The system performs preliminary actions by proactively identifying and rebuilding dependent worksheets before errors occur in user analyses. This prevents data inconsistency issues before they affect users, maintaining reliability without requiring complex manual tracking systems.
3Measurement precision
If dependent worksheets are automatically rebuilt with changes to model data sets, then error detection is improved, but processing time increases
Solution Approach 1:
The system applies partial action by selectively rebuilding only those dependent worksheets that are actually affected by changes to model data sets, rather than rebuilding all worksheets. This targeted approach maintains high error detection accuracy while minimizing unnecessary processing time for unaffected worksheets.
Data Source
AI summary
Tracking errors in data set lineage including receiving, from a user, a change to a model data set, wherein the model data set is a reusable modeling layer comprising at least a portion of a data source retrieved from a data warehouse; accessing a list of dependent worksheets utilizing the model data set as a data source wherein each dependent worksheet is configured to perform analysis on the portion of the first data source within the model data set without changing the model data set; generating an error report for the dependent worksheets utilizing the model data set; and providing, to the user, the error report for the dependent worksheets utilizing the model data set.