Provenance-Based Data Retention in Stream Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream-processing systems face challenges in efficiently determining the relevance of data elements to results produced and managing data retention for provenance queries, as existing methods assume all source data is available, which is impractical due to high data volumes and dynamic processing environments.

Innovation Solution

A system and method that uses provenance information to progressively trim stored data by computing a usage metric based on downstream recipients, discarding irrelevant data, and managing retention through a data usage manager configured with a provenance table to associate output ports with dependency functions, ensuring only relevant data is stored.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all source data is stored for provenance queries, then data availability for provenance is improved, but storage overhead increases significantly

Engineering Contradiction:

Improveprovenance data availabilityVSAvoidstorage overhead

Core Design Contradiction:

ReliabilityVSVolume of stationary object

Solution Approach 1:

The patent extracts only the necessary provenance information (usage metrics and dependency relationships) from the complete data stream, separating provenance tracking from full data storage. The system stores metadata about data usage and provenance dependencies rather than retaining all source data, thereby reducing storage overhead while maintaining provenance query capability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the data management approach by distinguishing between full data storage and provenance information storage. It maintains separate tracking of usage metrics and dependency graphs, allowing the system to retain only essential provenance metadata while discarding redundant source data, thus resolving the contradiction between provenance availability and storage cost.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If data is retained for provenance queries, then query accuracy is improved, but data management complexity increases

Engineering Contradiction:

Improveprovenance query accuracyVSAvoiddata management complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements feedback mechanisms where processing elements report usage metrics to the data management system, which then uses this information to dynamically manage data retention. The system continuously updates provenance information based on actual data consumption patterns, enabling accurate provenance queries while simplifying management through automated feedback-driven decisions.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system enables self-service data management by allowing processing elements to automatically report their own usage metrics and dependency relationships. The data management system autonomously processes this information to determine retention policies and manage provenance data, reducing the need for complex external management interventions.

Inventive Principle:

Principle #25Self-service

3Productivity

If usage metrics are computed for all data, then data relevance determination is improved, but processing overhead increases

Engineering Contradiction:

Improvedata relevance determination efficiencyVSAvoidprocessing overhead

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent applies partial action by computing usage metrics only for data that actually flows through processing elements and contributes to output results. Rather than analyzing all possible data, the system focuses computational resources on tracking usage of actively processed data, thereby determining data relevance efficiently without excessive processing overhead.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system performs preliminary action by pre-establishing dependency relationships and usage tracking mechanisms as data flows through the processing graph. Usage metrics are computed in advance during normal processing, so that when provenance queries are needed, the relevant information is already available, improving determination efficiency without adding significant overhead.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8856313B2Systems and methods for using provenance information for data retention in stream-processing

Publication Date: 2014.10.07 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A system and method for determining data usage based on provenance information, in a stream-processing system, includes progressively setting usage information for output stream data objects (SDOs), determining input SDOs that an output SDO depends on, based on a provenance dependency function; recursively feeding back the usage information for a subset of SDOs that can be discarded; and discarding the subset of SDOs. A system and method for data retention based on usage information, in a stream-processing system, includes managing retention of SDOs by deleting SDOs that are determined to be of null usage; and enhancing retention characteristics of SDOs that are deemed to have usage.