Distributed Data Processing Nodes for Privacy-Preserving Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site for analysis.

Innovation Solution

The system configures distributed applications to execute across multiple processing nodes associated with distinct data zones, performing computations locally and orchestrating execution to avoid data movement, thereby ensuring privacy and improving performance, security, and energy efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied to a centralized site for analysis, then data analytics can be performed, but privacy concerns arise and bandwidth consumption increases

Engineering Contradiction:

Improvedata analytics capabilityVSAvoidprivacy concerns

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

Instead of copying data to a centralized site for analysis, the patent inverts the approach by bringing computation to the distributed data nodes. Local processing is performed at each node, and only results are aggregated centrally, thereby eliminating privacy concerns while maintaining analytics capability

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the data processing task into local computations performed independently at each distributed node, with only intermediate results being transmitted. This segmentation eliminates the need to move raw data across the network, reducing bandwidth consumption and preserving privacy

Inventive Principle:

Principle #1Segmentation

2Productivity

If a shared distributed file system is deployed across multiple geographic locations, then data processing can be enabled, but system complexity increases and maintenance becomes difficult

Engineering Contradiction:

Improvedistributed data processing capabilityVSAvoidshared distributed file system configuration

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent extracts the file system dependency from the distributed processing architecture. Instead of requiring a shared distributed file system, each node processes data locally using its own local file system, eliminating the complexity of configuring and maintaining a shared file system across multiple locations

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies local quality by allowing each distributed node to operate with its own local file system characteristics and data formats. This eliminates the need for a unified shared file system while enabling distributed processing across heterogeneous environments

Inventive Principle:

Principle #3Local quality

3Productivity

If data is moved from local sites to a centralized site, then centralized analysis can be performed, but processing speed decreases and energy consumption increases

Engineering Contradiction:

Improvecentralized data analysisVSAvoidprocessing speed

Core Design Contradiction:

ProductivityVSSpeed

Solution Approach 1:

The patent inverts the traditional centralized processing model by performing computations locally at data nodes and aggregating results centrally. This eliminates data movement bottlenecks and significantly improves processing speed while reducing energy consumption

Inventive Principle:

Principle #13The other way round (Inversion)

4Adaptability or versatility

If a shared distributed file system is used across geographically dispersed sites, then data accessibility is enabled, but system maintenance becomes difficult

Engineering Contradiction:

Improvedata accessibility across locationsVSAvoidsystem maintenance

Core Design Contradiction:

Adaptability or versatilityVSEase of repair

Solution Approach 1:

The patent segments the system into independent nodes that each process data locally. This eliminates the shared file system component that requires maintenance, while maintaining data accessibility across geographically dispersed sites through distributed local processing

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Each node in the distributed system operates independently with its own local file system, making the system self-sufficient at each location. This eliminates the need for centralized file system maintenance while preserving data accessibility across sites

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10860622B1Scalable recursive computation for pattern identification across distributed data processing nodes

Publication Date: 2020.12.08 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to receive results of intermediate long-tail histogram computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network. The processing device is further configured to perform at least one global long-tail histogram computation based at least in part on the results of the intermediate long-tail histogram computations, and to utilize results of the intermediate and global long-tail histogram computations to identify patterns in the plurality of datasets. The distributed processing nodes are illustratively associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes. At least a subset of the receiving, performing and utilizing may be repeated in each of a plurality of iterations.