Feedback tuples for stream computing anomaly detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Stream computing environments face challenges in detecting and handling data flow anomalies, such as interruptions in data flow, which can go unnoticed for extended periods, especially in continuous 24/7 operations with minimal human oversight, due to the difficulty in distinguishing anomalies from normal fluctuations in data flow rates.

Innovation Solution

The implementation of feedback tuples in a stream computing environment to communicate flow status information upstream and downstream between stream operators, external data sources, and sinks, allowing for the detection, identification, and handling of data flow anomalies by monitoring deviations from expected flow rates and adaptive feedback mechanisms.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional database systems are used to store and analyze streaming data, then data storage and indexing capabilities are provided, but real-time processing efficiency and ability to handle large amounts of streaming data are insufficient

Engineering Contradiction:

Improvereal-time processing efficiencyVSAvoiddata flow anomaly detection capability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent implements feedback tuples that flow downstream through the stream processing pipeline, allowing each stream operator to report its processing status and detect anomalies. This feedback mechanism enables real-time monitoring of data flow health while maintaining high processing throughput, resolving the contradiction between productivity and reliability.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces feedback tuples as intermediary elements that carry diagnostic information through the data stream without interfering with the primary data processing flow. These tuples act as mediators between stream operators, enabling anomaly detection while preserving real-time processing efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If stream operators process data continuously at high speed, then real-time processing capability is improved, but the ability to detect data flow interruptions and anomalies is reduced

Engineering Contradiction:

Improvedata processing speedVSAvoiddata flow anomaly detection difficulty

Core Design Contradiction:

SpeedVSDifficulty of detecting and measuring

Solution Approach 1:

Downstream feedback tuples provide continuous status information about data flow health, enabling anomaly detection without slowing down the primary data processing stream. The feedback mechanism operates in parallel, maintaining high processing speed while improving detection capability.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system uses periodic feedback tuple generation and processing to monitor data flow status at regular intervals, enabling continuous anomaly detection while maintaining high overall processing speed through efficient batched feedback operations.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS11586519B2Feedback tuples for detecting data flow anomalies in stream computing environment

Publication Date: 2023.02.21 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Feedback tuples are implemented in a stream computing environment to provide data flow status information for use in detecting, identifying and/or handling data flow anomalies for a stream computing application.