Composite Data Product Update Rules for Fresh Dependency Builds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for managing dependencies in composite data products are inflexible, tightly coupled, and fail to ensure freshness and accuracy of data, leading to inefficiencies and modifiability issues.
Innovation Solution
A computer-implemented method that uses user-defined update rules and a centralized registry to determine when to trigger builds of downstream data products based on the latest builds and quality of upstream products, decoupling dependencies and enabling flexible, intelligent data updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If time-based triggers are used to initiate builds of data products, then builds can be scheduled at predefined frequencies, but the system fails to ensure data freshness when upstream builds take longer than anticipated or fail
Solution Approach 1:
The system implements feedback mechanisms where downstream data products continuously monitor the build status and freshness of their upstream dependencies. When upstream builds complete or fail, this information is fed back to downstream products, which then autonomously decide whether to trigger their own builds based on predefined freshness thresholds and dependency status, rather than relying on rigid scheduled triggers.
Solution Approach 2:
Downstream data products are empowered to self-manage their own build triggering based on their dependency status. Each downstream product monitors its upstream dependencies and autonomously determines when to initiate a build based on whether the upstream data meets its freshness requirements, eliminating the need for centralized scheduled triggers and enabling adaptive, reliability-driven build initiation.
2Reliability
If upstream data products track and directly trigger downstream consumer processes, then data freshness can be maintained, but the system becomes tightly coupled and impossible to modify when adding new data products
Solution Approach 1:
The patent introduces a centralized registry as an intermediary layer between upstream and downstream data products. Instead of direct tracking and triggering relationships, upstream products publish their build information and dependency definitions to the registry, while downstream products consume this information from the registry to make build decisions. This decoupling mechanism maintains data freshness propagation while eliminating tight coupling and reducing system complexity.
Solution Approach 2:
The system segments the dependency management function into independent, loosely-coupled components: upstream products define their dependencies in configuration files, the registry stores and manages this information separately, and downstream products consume information from the registry. This segmentation allows each component to be modified independently, enabling easy addition of new data products without requiring changes to existing upstream products.
3Reliability
If careful time-based coordination is implemented for builds in dependency chains, then data freshness can be maintained, but the system becomes inflexible and cannot adapt when builds take longer than anticipated
Solution Approach 1:
The system replaces static, pre-scheduled build triggers with dynamic, condition-based triggering. Downstream data products continuously evaluate their upstream dependency status and autonomously decide when to trigger builds based on current freshness requirements and actual upstream build completion times. This dynamic approach allows the system to adapt to varying build durations and maintain data accuracy without rigid time-based coordination.
Data Source
Figure 1
Figure 2
Figure 3a~3b
AI summary
This disclosure relates to methods, devices, and computer-readable media relating to update rules for composite data products (i.e., data products which depend on one or more upstream data products). One such method comprises receiving a data product definition for the downstream data product, wherein the data product definition identifies one or more upstream data products on which the downstream data product depends, receiving data indicating the latest build for each upstream data product, determining, based on the received data and one or more user-defined update rules for the downstream data product, whether an update condition is satisfied, and in accordance with a determination that the update condition is satisfied, triggering a build of the downstream data product.