Distributed Data Processing Using Local Computational Frameworks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site for analysis.
Innovation Solution
The system distributes computations across multiple data processing clusters using distinct computational frameworks, allowing each cluster to process data locally with its own framework and combining results through a global data structure, eliminating the need for a shared file system and preserving data privacy by keeping raw data local.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is copied to a centralized site for analysis, then data analytics can be performed, but data privacy is compromised and transfer time increases
Solution Approach 1:
Instead of moving data to a centralized site for analysis, the patent inverts the approach by bringing the computational framework to the distributed data locations. Each data zone runs its own computational framework locally, performing analytics in-place without requiring data transfer to a central location.
Solution Approach 2:
The patent segments the computational framework into distributed instances that can operate independently across multiple data zones. Each zone maintains its own framework instance, enabling parallel processing of data locally while contributing to global analytics results through coordinated computation.
2Productivity
If a shared distributed file system is deployed across multiple geographic locations, then data processing can be distributed, but system complexity and maintenance difficulty increase
Solution Approach 1:
The patent extracts the file system dependency from the distributed processing architecture. Instead of requiring a shared distributed file system, each data zone maintains its own local storage, and the computational framework operates independently on local data without needing centralized file system management.
Solution Approach 2:
Each data zone is configured with its own local computational framework instance and local storage, allowing each location to operate independently with its own data processing capabilities. This local autonomy eliminates the need for complex shared file system configuration while maintaining distributed processing efficiency.
3Adaptability or versatility
If a shared distributed file system is deployed across multiple entities and geographic locations, then data can be accessed centrally, but privacy concerns and access control complexity increase
Solution Approach 1:
The patent segments data access authority by maintaining separate data zones for different entities, with each zone's computational framework operating independently on its own data. This segmentation ensures that data remains within organizational boundaries while still enabling distributed collaborative analytics through framework coordination.
Solution Approach 2:
The computational framework acts as an intermediary that enables data processing without direct data sharing. The framework coordinates computation across data zones and aggregates results, allowing collaborative analytics while keeping raw data confined to its original ownership boundaries and eliminating privacy concerns associated with data copying.
Data Source
AI summary
An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed computations from respective ones of the data processing clusters. Each of the data processing clusters is configured to process data from a data source of the corresponding data zone using a local data structure and an associated computational framework of that data processing cluster. A first one of data processing clusters utilizes a first local data structure configured to support a first computational framework, and at least a second one of the data processing clusters utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.


