Data Pipeline Tracking Stamps for Incident Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing computing systems face challenges in monitoring and alerting mechanisms for data pipelines, leading to negative impacts on downstream processes and increased costs due to incidents caused by software changes or other issues.

Innovation Solution

A method and system for tracking data objects in a data pipeline by assigning a record number, creating tracking stamps, and broadcasting them to a record tracking queue, which updates records with station arrays, station codes, and timestamp arrays to track the data object's progression through the pipeline.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data pipelines process millions of data points in real-time without adequate monitoring, then processing speed is maintained, but incidents go undetected causing negative impacts on downstream processes

Engineering Contradiction:

Improvedata processing speedVSAvoidincident detection capability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent introduces tracking stamps as intermediary elements that are embedded with data objects as they flow through the data pipeline. These stamps serve as mediators that carry identification information without interfering with the actual data processing, enabling monitoring while maintaining processing speed

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system creates tracking stamps that are copies or representations of data object identifiers. These stamps are broadcast to monitoring systems independently of the main data flow, allowing incident detection without duplicating the full data processing workload

Inventive Principle:

Principle #26Copying

2Reliability

If comprehensive monitoring and alerting mechanisms are implemented to track data objects, then incident detection capability is improved, but system complexity increases

Engineering Contradiction:

Improveincident detection capabilityVSAvoidmonitoring system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The monitoring function is segmented into independent components: tracking stamp generation, stamp broadcasting, and stamp processing. Each component operates independently, allowing the monitoring system to be implemented incrementally and maintained separately from the core data pipeline

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The tracking stamp mechanism serves multiple functions simultaneously: it identifies data objects, tracks their progression through the pipeline, enables incident detection, and provides audit trails. This multi-functionality reduces the need for separate monitoring mechanisms

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Loss of time

If tracking stamps are broadcast to monitor data object progression through the pipeline, then incident identification speed is improved, but network communication overhead increases

Engineering Contradiction:

Improveincident identification timeVSAvoidnetwork communication overhead

Core Design Contradiction:

Loss of timeVSLoss of energy

Solution Approach 1:

Tracking stamps are broadcast selectively to specific monitoring channels or subscribers based on their location and function in the pipeline. Not all components receive all tracking stamps, reducing unnecessary network traffic while maintaining effective monitoring coverage

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system broadcasts tracking stamps with sufficient information for identification and tracking, but not necessarily all possible attributes of the data object. This partial action provides adequate monitoring capability while minimizing the data volume transmitted over the network

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250110936A1Systems and methods for tracking of data records

Publication Date: 2025.04.03 FIDELITY INFORMATION SERVICES LLC

AI summary

A computer-implemented method for tracking a data object in a data pipeline. The method may include receiving a data object from an data source; assigning a record number to the data object; determining a first tracking stamp for the data object; broadcasting the first tracking stamp to a channel of a record tracking queue; initiating a record in the record tracking queue for the data object, the record including the record number, a station array, a station code, and a timestamp array, wherein the first tracking stamp updates the station array, station code, and timestamp array; transferring the data object to a second component; determining a second tracking stamp for the data object, the second tracking stamp including the record number; broadcasting the second tracking stamp to the channel; and updating the record with the second tracking stamp to indicate that the data object was received by the second component.