Distributed In-Memory Computation Across Data Zones

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing distributed data processing frameworks face challenges in handling geographically dispersed data across multiple sites with different ownership and formats, requiring shared distributed file systems that are difficult to configure and maintain, leading to inefficiencies and privacy concerns when copying data to a centralized site for analytics.

Innovation Solution

The system distributes in-memory computations across multiple data processing clusters with local data zones, using a recursive approach to perform computations locally within each cluster, aggregating results without the need for a shared file system, and utilizing a distributed catalog service to manage data resources and preserve privacy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is copied from local sites to a centralized site for analytics, then data accessibility for processing is improved, but data transfer time and network bandwidth consumption increase significantly

Engineering Contradiction:

Improvedata accessibilityVSAvoiddata transfer time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Instead of moving data to a centralized location for processing, the system inverts the approach by bringing computation to where the data resides. Multiple distributed processing nodes execute analytics tasks locally at geographically dispersed sites, eliminating the need to copy data across networks while maintaining processing capabilities.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The centralized analytics task is segmented into multiple distributed sub-tasks that can be executed independently across different geographic locations. Each processing node handles a portion of the computation locally, and results are aggregated to form the complete analytics output, avoiding centralized data collection.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data sharing and centralized processing are enabled, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata sharing capabilityVSAvoidfile system configuration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent extracts the file system dependency from the distributed processing architecture. Instead of requiring a shared distributed file system, each processing node accesses data through local file systems or data sources directly at its location, eliminating the need for complex centralized file system configuration and maintenance across geographic boundaries.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Each processing node is configured with local data access capabilities tailored to its specific geographic location and data sources. Rather than imposing a uniform shared file system architecture, the system allows each site to maintain its own data access methods, reducing overall system complexity while enabling data sharing through standardized processing interfaces.

Inventive Principle:

Principle #3Local quality

3Productivity

If data is copied to a centralized site for analytics, then processing efficiency is improved, but data privacy and security concerns are exacerbated

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidprivacy concerns

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

The system inverts the traditional analytics model by keeping data in place and bringing computation to the data. This prevents data from being copied or transmitted to centralized locations, thereby maintaining privacy and security while still enabling efficient distributed processing and aggregation of results.

Inventive Principle:

Principle #13The other way round (Inversion)

Data Source

PatentUS10656861B1Scalable distributed in-memory computation

Publication Date: 2020.05.19 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to distribute in-memory computations across at least first and second nodes of respective distinct data processing clusters of a plurality of data processing clusters over at least one network, and to aggregate results of the distributed in-memory computations for delivery to a requesting client device. The data processing clusters are associated with respective distinct data zones, and the first and second nodes of the respective distinct data processing clusters are configured to perform corresponding portions of the distributed in-memory computations utilizing respective ones of first and second in-memory datasets locally accessible within their respective data zones. The in-memory computations in some embodiments illustratively comprise Spark computations, such as Spark Core batch computations. The in-memory datasets in such an arrangement may comprise respective Spark resilient distributed datasets.