Data Set Lineage Metadata for Selective Update Propagation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Managing large volumes of data sets for machine learning is time-consuming and error-prone, requiring significant effort to maintain and update data sets across numerous data sets.

Innovation Solution

Implementing data set management using data set lineage metadata to automatically propagate data modifications to any number of different groups of data, reducing the time and computing/storage resources needed for maintaining and updating data sets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data sets are manually managed and updated, then data quality can be maintained, but significant time and human effort are required

Engineering Contradiction:

Improvedata qualityVSAvoidtime to manage and update data sets

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system enables self-service data propagation where lineage metadata automatically triggers and executes data transformation operations. When source data changes, the system autonomously identifies affected derived data sets and propagates updates without requiring manual intervention, thus maintaining data quality while eliminating time-consuming manual management

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system implements feedback mechanisms through lineage metadata that tracks data dependencies. This feedback loop allows the system to automatically detect when source data changes and identify which derived data sets need updating, enabling automated quality maintenance through continuous monitoring and propagation

Inventive Principle:

Principle #23Feedback

2Stability of the object's composition

If data modifications are propagated to all derived data sets, then data consistency is maintained, but computing and storage resources are consumed

Engineering Contradiction:

Improvedata consistencyVSAvoidcomputing and storage resources

Core Design Contradiction:

Stability of the object's compositionVSUse of energy by moving object

Solution Approach 1:

The system segments the data propagation process by using lineage metadata to identify and update only the specific derived data sets that are actually affected by source data changes. This selective approach maintains data consistency while avoiding unnecessary computing and storage resources that would be consumed by propagating changes to all data sets

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies local quality by tailoring the propagation scope to the specific impact of each data modification. Lineage metadata enables the system to apply updates locally to only those data sets that have direct dependencies on the changed source data, rather than uniformly propagating to all derived data sets, thus optimizing resource usage while maintaining consistency

Inventive Principle:

Principle #3Local quality

3Adaptability or versatility

If multiple data sets are managed independently, then data autonomy is maintained, but error propagation across data sets increases

Engineering Contradiction:

Improvedata autonomyVSAvoiderror propagation

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system implements universality through a unified lineage metadata framework that works across all data sets regardless of their independence. This universal metadata structure enables the system to maintain data autonomy while simultaneously providing cross-data-set error detection and propagation prevention through centralized lineage tracking

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Loss of information

If comprehensive data tracking is implemented, then data lineage visibility is improved, but system complexity increases

Engineering Contradiction:

Improvedata lineage visibilityVSAvoidsystem complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The system applies preliminary action by capturing and storing lineage metadata at the time of data creation and transformation. This advance recording of data relationships provides comprehensive lineage visibility without requiring complex real-time tracking systems, thus reducing overall system complexity while maintaining complete data lineage information

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12380062B1Data set management using data set lineage metadata

Publication Date: 2025.08.05 AMAZON TECH INC

AI summary

A data set management of a provider network may allow a user to create new data set instances. When a data set instance is created, data set lineage metadata is also generated to describe the new data set instance, including the transformation that was applied to data in order to create the data set instance. When modifications are made to source data (e.g., a data bucket), then the modifications are propagated via transformations to the parent data set instance and to any child data set instances according to the data set lineage metadata in order to update the data set instances. When modifications are made to a parent data set instance to create an updated parent data set instance, then the modifications are propagated via transformations to any child data set instances according to the data set lineage metadata. Transformations and transformation patterns may also be defined and scheduled.