Distributed In-Memory Computation via Local Batch Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations without a shared distributed file system, leading to inefficiencies and privacy concerns due to the need for data copying and centralized data analytics.

Innovation Solution

The implementation of a multi-cluster distributed data processing platform that distributes in-memory computations across multiple data processing clusters using batch mode extensions, allowing local data structures to perform computations and combine results without requiring a shared file system, utilizing frameworks like Spark Core, Spark SQL, Spark MLlib, and Spark GraphX, and leveraging a WWH catalog for data resource management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied from local sites to a centralized site for data analytics, then data processing can be performed, but data transfer time increases and privacy concerns arise

Engineering Contradiction:

Improvedata processing capabilityVSAvoiddata transfer time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

Instead of copying data to a centralized location for processing, the system inverts the approach by bringing the processing capability to the distributed data locations. Multiple local sites perform data analytics in place using their own computational resources, eliminating the need for data transfer while maintaining processing capability.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The centralized data processing task is segmented into multiple local processing tasks distributed across different sites. Each local site independently processes its own data using local computational resources, and results are subsequently aggregated, eliminating data transfer time while maintaining overall processing productivity.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data accessibility is improved, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata accessibilityVSAvoidfile system configuration and maintenance

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Each geographic location maintains its own local file system with local data storage and processing capabilities. Sites access their own local data resources independently without requiring a unified shared file system, thereby maintaining data accessibility while avoiding the complexity of cross-geographic file system synchronization and maintenance.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

A catalog service acts as an intermediary that provides a unified view of distributed data resources across multiple locations without requiring actual shared storage. The catalog enables data accessibility by indexing and describing local data resources, allowing queries to be routed to appropriate local sites while avoiding direct file system sharing complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If data is copied to a centralized site for analytics, then processing can be performed, but privacy and security concerns are raised

Engineering Contradiction:

Improvedata analytics capabilityVSAvoidprivacy concerns

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

Instead of centralizing data for processing, the system inverts the approach by distributing processing capabilities to where the data resides. Local sites perform analytics on their own data using local computational resources, eliminating privacy risks associated with data copying while maintaining full analytics capability.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The centralized analytics task is segmented into multiple independent local analytics operations. Each site processes its own data locally, and only aggregated results (not raw data) are shared, thereby maintaining analytics productivity while protecting data privacy through spatial segmentation of processing operations.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10331380B1Scalable distributed in-memory computation utilizing batch mode extensions

Publication Date: 2019.06.25 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to distribute in-memory computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed in-memory computations from the data processing clusters. The distributed in-memory computations utilize local data structures of respective ones of the data processing clusters. A given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters. The local data structures are configured to support batch mode extensions such as Spark SQL, Spark MLlib or Spark GraphX for performance of the distributed in-memory computations.