Inferred Job Dependencies for Analytics Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Analytics systems face challenges in ensuring that data is available and up-to-date before it is consumed by processing jobs, leading to delays or incorrect reports if the source data is not properly managed.

Innovation Solution

The system allows analysts to define dependencies on data objects rather than jobs, enabling the automatic determination of source jobs responsible for generating the required data, ensuring that these jobs complete before the consuming jobs are executed.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If analysts manually specify job dependencies, then data availability can be ensured, but the complexity of job management increases significantly

Engineering Contradiction:

Improvedata availabilityVSAvoidjob management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system enables self-service by automatically inferring job dependencies from data object relationships. Analysts only need to specify data object dependencies, while the system autonomously determines the corresponding job dependencies, eliminating manual intervention and reducing management complexity.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

Data objects serve as intermediaries between jobs. Instead of directly managing complex job-to-job dependencies, analysts work with simpler data object dependencies. The system uses these data object relationships as a mediator to automatically derive the necessary job scheduling information.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If the system infers dependencies automatically from data objects, then job management is simplified, but the system complexity increases

Engineering Contradiction:

Improvejob dependency specificationVSAvoidsystem complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

Data objects act as intermediaries that simplify the user interface. Analysts interact with familiar data object concepts rather than complex job dependency graphs. The system leverages this intermediary layer to automatically compute job dependencies, making the system easier to operate despite increased internal processing requirements.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If source jobs are ensured to complete before sink jobs, then data integrity is maintained, but processing time increases due to waiting

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by inferring and establishing job dependencies in advance based on data object relationships. This allows the scheduling system to proactively plan job execution order, ensuring source jobs are scheduled to complete before sink jobs without ad-hoc waiting, thus maintaining data integrity while optimizing timing.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10372492B2Job-processing systems and methods with inferred dependencies between jobs

Publication Date: 2019.08.06 DROPBOX INC

AI summary

An analytics system that executes processing jobs infers dependencies between jobs to be executed based on identification of dependencies between a “sink” job and a source data object on which the sink job depends. Given a job definition for the sink job that identifies a source data object, the system can identify a “source” job that produces the source data object and can infer a dependency of the sink job on the source job. The system can schedule executions of the source and sink jobs such that the source job completes (or completes generation of the source data object) before the sink job is launched.