Distributed Data Processing Using Local Computational Frameworks

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site for analysis.

Innovation Solution

The system distributes computations across multiple data processing clusters using distinct computational frameworks, allowing each cluster to process data locally with its own framework and combining results through a global data structure, eliminating the need for a shared file system and preserving data privacy by keeping raw data local.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied to a centralized site for analysis, then data analytics can be performed, but data privacy is compromised and transfer time increases

Engineering Contradiction:
Improvedata analytics capabilityVSAvoiddata transfer time
Core Design Contradiction:
ProductivityVSLoss of time

Solution Approach 1:

Instead of moving data to a centralized site for analysis, the patent inverts the approach by bringing the computational framework to the distributed data locations. Each data zone runs its own computational framework locally, performing analytics in-place without requiring data transfer to a central location.

Inventive Principle:
Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the computational framework into distributed instances that can operate independently across multiple data zones. Each zone maintains its own framework instance, enabling parallel processing of data locally while contributing to global analytics results through coordinated computation.

Inventive Principle:
Principle #1Segmentation

2Productivity

If a shared distributed file system is deployed across multiple geographic locations, then data processing can be distributed, but system complexity and maintenance difficulty increase

Engineering Contradiction:
Improvedistributed data processing capabilityVSAvoidfile system configuration and maintenance
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The patent extracts the file system dependency from the distributed processing architecture. Instead of requiring a shared distributed file system, each data zone maintains its own local storage, and the computational framework operates independently on local data without needing centralized file system management.

Inventive Principle:
Principle #2Taking out (Extraction)

Solution Approach 2:

Each data zone is configured with its own local computational framework instance and local storage, allowing each location to operate independently with its own data processing capabilities. This local autonomy eliminates the need for complex shared file system configuration while maintaining distributed processing efficiency.

Inventive Principle:
Principle #3Local quality

3Adaptability or versatility

If a shared distributed file system is deployed across multiple entities and geographic locations, then data can be accessed centrally, but privacy concerns and access control complexity increase

Engineering Contradiction:
Improvedata accessibilityVSAvoidprivacy concerns
Core Design Contradiction:
Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

The patent segments data access authority by maintaining separate data zones for different entities, with each zone's computational framework operating independently on its own data. This segmentation ensures that data remains within organizational boundaries while still enabling distributed collaborative analytics through framework coordination.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The computational framework acts as an intermediary that enables data processing without direct data sharing. The framework coordinates computation across data zones and aggregates results, allowing collaborative analytics while keeping raw data confined to its original ownership boundaries and eliminating privacy concerns associated with data copying.

Inventive Principle:
Principle #24Intermediary (Mediator)

Data Source

PatentUS10776404B2Scalable distributed computations utilizing multiple distinct computational frameworks
Publication Date: 2020.09.15 EMC IP HLDG CO LLC
  • US10776404B2 patent drawing
  • US10776404B2 patent drawing
  • US10776404B2 patent drawing

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed computations from respective ones of the data processing clusters. Each of the data processing clusters is configured to process data from a data source of the corresponding data zone using a local data structure and an associated computational framework of that data processing cluster. A first one of data processing clusters utilizes a first local data structure configured to support a first computational framework, and at least a second one of the data processing clusters utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.