Distributed Analytics Platform for Privacy-Preserving Data Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations without a shared distributed file system, leading to inefficiencies and privacy concerns due to the need for data copying and centralized data analytics.

Innovation Solution

A multi-cluster distributed data processing platform that enables analytics platforms to facilitate distributed computations across multiple data processing clusters without sharing datasets, using a global insight exchange platform to register datasets and analytic services, allowing local processing within each cluster and synchronization of results for global insights.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is copied from local sites to a centralized site for data analytics, then data analytics can be performed, but data transfer time increases and privacy concerns arise

Engineering Contradiction:

Improvedata analytics capabilityVSAvoiddata transfer time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Instead of copying data to a centralized location for analysis, the patent inverts the approach by bringing the analysis capability to the distributed data locations. Multiple local sites perform analytics on their own data and share only the results, eliminating the need to transfer large volumes of raw data while maintaining analytics capability.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the centralized analytics process into distributed local analytics processes. Each local site independently performs analytics on its own data segment, and the results are aggregated. This segmentation eliminates the bottleneck of centralized data collection and reduces overall data transfer requirements.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data accessibility is improved, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata accessibilityVSAvoidsystem configuration complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary layer (gateway or interface) that allows local sites to access and share data without requiring a fully integrated shared file system. This intermediary handles data exchange protocols and formats, providing accessibility while shielding local systems from the complexity of direct integration.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent creates a universal interface or gateway that can work with multiple different local file systems and data formats. This universal layer provides data accessibility across diverse systems without requiring each system to adopt a specific complex configuration, thereby reducing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If data is centralized for analytics processing, then computation efficiency is improved, but data privacy and security concerns worsen

Engineering Contradiction:

Improvecomputation efficiencyVSAvoidprivacy concerns

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

The patent inverts the traditional centralized processing model by performing computation at the distributed data locations rather than collecting data centrally. Local sites execute analytics on their own data, maintaining privacy while achieving computational efficiency through parallel processing across multiple sites.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent extracts only the necessary analytics results from local data rather than transferring the entire datasets. By taking out only the computed insights and sharing those centrally, the system maintains computation efficiency while minimizing privacy risks associated with data exposure.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS10496926B2Analytics platform for scalable distributed computations

Publication Date: 2019.12.03 EMC IP HLDG CO LLC

AI summary

An analytics platform is configured to provide at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones. The analytics platform is further configured to maintain one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations, and to initiate execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services. The analytics platform allocates to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations. The analytic services are illustratively executed on the datasets without the datasets leaving their respective data zones.