Distributed Data Processing Clusters Using Distinct Computational Frameworks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site for analysis.

Innovation Solution

The system distributes computations across multiple data processing clusters using distinct computational frameworks, allowing each cluster to process data locally with its own framework and structure, and combines results using a global data structure, eliminating the need for a shared file system and preserving data privacy by keeping raw data local.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is copied to a centralized site for analysis, then data analytics can be performed, but data transfer time increases and privacy concerns arise

Engineering Contradiction:

Improvedata analytics capabilityVSAvoiddata transfer time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Instead of copying data to a centralized site for analysis, the patent inverts the approach by bringing the computational framework to the distributed data locations. Each data processing cluster executes local computations on its own data, eliminating the need for data transfer while maintaining analytics capability.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the centralized analytics process into distributed computational tasks executed across multiple independent data processing clusters. Each cluster handles its own data locally, dividing the overall analytics workload into parallel segments that operate independently without requiring data centralization.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data accessibility is improved, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata accessibilityVSAvoidsystem configuration and maintenance complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent applies local quality by allowing each data processing cluster to maintain its own local file system with characteristics optimized for its specific needs and data types. Instead of enforcing a uniform shared file system across all locations, each cluster adapts its file system implementation to local requirements, reducing complexity while maintaining accessibility.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The computational framework is designed to be universal and framework-agnostic, capable of operating with different file system implementations across various clusters. This multi-functionality allows the system to work with diverse local file systems without requiring a standardized shared file system, thereby reducing configuration and maintenance complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If different data processing clusters use different computational frameworks, then framework-specific optimizations can be applied, but integration complexity increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidframework integration complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary layer that translates between different computational frameworks and a unified execution model. This mediator enables clusters using different frameworks (e.g., MapReduce, Spark, Flink) to participate in distributed computations without direct integration between them, maintaining processing efficiency while reducing integration complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent resolves framework integration complexity by transitioning from a horizontal integration approach (directly connecting different frameworks) to a vertical dimension approach. Each framework operates independently at its own level, with the coordination and result aggregation happening at a higher abstraction dimension, thereby simplifying integration while preserving framework-specific optimizations.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS10366111B1Scalable distributed computations utilizing multiple distinct computational frameworks

Publication Date: 2019.07.30 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed computations from respective ones of the data processing clusters. Each of the data processing clusters is configured to process data from a data source of the corresponding data zone using a local data structure and an associated computational framework of that data processing cluster. A first one of data processing clusters utilizes a first local data structure configured to support a first computational framework, and at least a second one of the data processing clusters utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.