Stream Dependency Explorer for Real-Time Big Data Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems face challenges in interactively exploring and analyzing dependencies in streaming data due to the high volume, velocity, and variability of Big Data, particularly in real-time environments, where traditional data processing methods are inadequate for handling multidimensional data streams.

Innovation Solution

The Stream Dependency Explorer (SDE) employs regression models to compute and visualize attribute dependencies within multidimensional data streams, allowing for continuous analysis and visualization of dependencies, detection of changing structures, and recommendation of significant exploration steps, while supporting both online and post-hoc analysis through the complex event processing paradigm.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional data processing methods are used on streaming data, then system simplicity is maintained, but the system cannot handle high volume and velocity of Big Data effectively

Engineering Contradiction:

Improvedata processing capabilityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments the complex task of streaming data analysis into multiple specialized components: a stream processor for real-time event processing, a visual analytics engine for dependency detection, and a model store for statistical model management. Each component handles specific aspects of data processing, enabling the system to manage high-volume streaming data effectively while maintaining modularity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an event bus as an intermediary component that facilitates communication between data sources, stream processors, and visual analytics engines. This mediator abstracts the complexity of data flow management, allowing components to interact through standardized event interfaces without direct coupling, thus handling high-velocity data streams efficiently.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If comprehensive dependency analysis is performed on multidimensional data streams, then measurement precision of relationships is improved, but processing time increases

Engineering Contradiction:

Improvedependency detection accuracyVSAvoidanalysis time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system pre-computes and stores statistical models (e.g., regression models, correlation matrices) in a model store before they are needed for analysis. When dependency analysis is requested, the visual analytics engine retrieves pre-prepared models and applies them to the streaming data, significantly reducing real-time processing time while maintaining high measurement precision for dependency detection.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements selective dependency analysis by allowing users to specify particular attributes or time windows for analysis rather than processing the entire multidimensional dataset uniformly. The visual analytics engine focuses computational resources on locally relevant dependencies based on user interests and detected anomalies, improving measurement precision for critical relationships while reducing overall processing time.

Inventive Principle:

Principle #3Local quality

3Loss of information

If real-time visualization of streaming data dependencies is provided, then information completeness is improved, but system resource consumption increases

Engineering Contradiction:

Improveinformation completenessVSAvoidcomputational resource consumption

Core Design Contradiction:

Loss of informationVSUse of energy by moving object

Solution Approach 1:

The system extracts and visualizes only the most significant dependencies and patterns from the multidimensional streaming data, rather than displaying all available information. The visual analytics engine identifies key relationships using statistical models and presents them in optimized visualizations, maintaining information completeness for critical insights while reducing computational resource consumption by excluding redundant data processing and display elements.

Inventive Principle:

Principle #2Taking out (Extraction)

4Ease of operation

If interactive exploration capabilities are added to the data analysis system, then ease of operation is improved, but device complexity increases

Engineering Contradiction:

Improveuser interaction capabilityVSAvoidsystem architecture complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The visual analytics system provides automated dependency detection and visualization generation, reducing the need for complex manual configuration. Users can interactively explore data by selecting attributes of interest, and the system automatically retrieves relevant pre-computed models, performs analysis, and generates visualizations without requiring users to understand underlying computational complexity, thus improving ease of operation while managing system complexity through automation.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS9792259B2Systems and/or methods for interactive exploration of dependencies in streaming data

Publication Date: 2017.10.17 CUMULOCITY

AI summary

Certain example embodiments allow for the continuous discovery and analysis of dependencies in a multidimensional event stream in which various regression models are continuously modified in real time and regression models are switched, if necessary or desirable. The event streams in certain example embodiments are continuously arriving Big Data streams of heterogeneous types, and the discovery and analysis is done dynamically in real time.