Distributed Data Streaming Across Clusters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site for analysis.

Innovation Solution

The system distributes data streaming computations across multiple data processing clusters, using a global computation graph to process data batches independently and asynchronously, with a WWH catalog managing data resources and computations to maintain privacy and efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied from local sites to a centralized site for analysis, then data processing can be performed, but processing speed decreases and privacy concerns arise

Engineering Contradiction:

Improvedata processing efficiencyVSAvoiddata transfer time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

Instead of moving data to a centralized location for processing, the patent inverts the approach by bringing processing capabilities to distributed locations where data resides. Multiple local data processing clusters perform computations locally, and only results are aggregated, eliminating data transfer bottlenecks and privacy concerns.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the centralized processing task into distributed processing across multiple local clusters. Each cluster independently processes its local data, and results are combined through a coordination layer, achieving parallel processing without data centralization.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data accessibility is improved, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata accessibilityVSAvoidfile system configuration and maintenance

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent extracts the file system dependency from the distributed processing architecture. Instead of requiring a shared distributed file system across clusters, each cluster maintains its own local file system, and the system uses data location metadata and result aggregation to achieve global data accessibility without centralized file management complexity.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If data is copied to a centralized site, then unified analysis is enabled, but data privacy and governance compliance are compromised

Engineering Contradiction:

Improveunified data analysis capabilityVSAvoidprivacy concerns

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

The patent inverts the traditional centralized analysis model by performing unified analysis through distributed computation. Processing logic is coordinated across clusters to achieve unified analytical results while data remains localized, preventing privacy violations and enabling compliance with data governance regulations.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent introduces a coordination layer that acts as an intermediary between local clusters. This layer manages the distributed computation graph, coordinates data flow, and aggregates results without requiring actual data movement, thus maintaining privacy while enabling unified analysis.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10404787B1Scalable distributed data streaming computations across multiple data processing clusters

Publication Date: 2019.09.03 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed data streaming computations across data processing clusters associated with respective data zones, and in each of the data processing clusters, to separate a data stream provided by a data source of the corresponding data zone into a plurality of data batches and process the data batches to generate respective result batches. Multiple ones of the data batches across the data processing clusters are associated with a global data batch data structure, and multiple ones of the result batches across the data processing clusters are associated with a global result batch data structure based at least in part on the global data batch data structure. The result batches are processed in accordance with the global result batch data structure to generate one or more global result streams.