Composite Data Product Updates via Registry-Based Build Checks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for managing dependencies in composite data products are inflexible, tightly coupled, and fail to ensure freshness and accuracy of data, leading to inefficiencies and complexity in data product updates.

Innovation Solution

A computer-implemented method that uses user-defined update rules and a centralized registry to determine when downstream data products should be built, based on the latest build status of upstream products, without requiring upstream products to track downstream consumers, allowing for flexible and efficient data updates.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Extent of automation

If time-based triggers are used to initiate builds of data products, then build scheduling becomes automated, but the system fails to ensure data freshness when upstream builds take longer than anticipated or fail

Engineering Contradiction:

Improvebuild scheduling automationVSAvoiddata freshness guarantee

Core Design Contradiction:

Extent of automationVSReliability

Solution Approach 1:

The system implements feedback by having downstream data products query the registry to check the actual build status and timestamp of upstream data products before initiating their own builds. This feedback loop ensures that builds are triggered based on real upstream data availability rather than predetermined schedules, resolving the contradiction between automation and reliability.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary action by checking upstream data product build status before triggering downstream builds. The registry stores build information in advance, allowing downstream products to verify data freshness conditions are met before commencing their build process, ensuring data freshness is guaranteed.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If upstream data products track all downstream consumers to trigger builds, then data freshness is maintained, but system complexity and coupling increase significantly

Engineering Contradiction:

Improvedata freshnessVSAvoiddependency tracking complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system inverts the traditional dependency tracking approach by having downstream data products query upstream build status from a centralized registry, rather than having upstream products track and notify all downstream consumers. This reversal eliminates the need for upstream products to maintain complex lists of dependents while still ensuring data freshness.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The centralized registry acts as an intermediary between upstream and downstream data products. It stores build information and allows downstream products to query upstream status without direct tracking relationships, reducing system complexity and coupling while maintaining data freshness guarantees.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Extent of automation

If blind builds are triggered by scheduled triggers, then automation is simplified, but computational resources are wasted when upstream data is stale or builds fail

Engineering Contradiction:

Improvebuild trigger simplicityVSAvoidcomputational resource waste

Core Design Contradiction:

Extent of automationVSLoss of energy

Solution Approach 1:

The system performs preliminary verification by querying the registry to check upstream data product build status and timestamps before triggering downstream builds. This preliminary check prevents wasted computational resources on builds that would produce stale or invalid data, while maintaining automated triggering through configurable rules.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Downstream data products autonomously determine whether their build conditions are met by querying the registry and evaluating their own configurable trigger rules. This self-service approach eliminates the need for complex centralized scheduling while preventing resource waste through intelligent condition checking.

Inventive Principle:

Principle #25Self-service

4Adaptability or versatility

If user-defined update rules are implemented, then flexibility and adaptability improve, but system complexity in rule evaluation increases

Engineering Contradiction:

Improveupdate rule flexibilityVSAvoidrule evaluation complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system extracts the complexity of rule evaluation from the core data product build logic and implements it as separate, configurable trigger rules defined by users. These rules are evaluated independently based on registry data, allowing flexibility in update logic without complicating the fundamental build process.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20250342149A1Configurable update rules for composite data products

Publication Date: 2025.11.06 FAIR ISAAC & CO INC

AI summary

This disclosure relates to methods, devices, and computer-readable media relating to update rules for composite data products (i.e., data products which depend on one or more upstream data products). One such method comprises receiving a data product definition for the downstream data product, wherein the data product definition identifies one or more upstream data products on which the downstream data product depends, receiving data indicating the latest build for each upstream data product, determining, based on the received data and one or more user-defined update rules for the downstream data product, whether an update condition is satisfied, and in accordance with a determination that the update condition is satisfied, triggering a build of the downstream data product.