Distributed Catalog Service for Privacy-Preserving Data Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current distributed data processing frameworks face challenges in efficiently processing data across multiple geographic locations due to the need for a shared distributed file system, which is difficult to configure and maintain, and raises privacy concerns when data is copied to a centralized site.

Innovation Solution

The implementation of a distributed catalog service that orchestrates the execution of applications across multiple processing nodes, allowing computations to be performed locally within each data zone, using a layered extensible data model and microservices to abstract and manage resources, thereby eliminating the need for data movement and ensuring privacy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is copied from local sites to a centralized site for analytics, then centralized data processing can be performed, but data privacy is compromised and bandwidth is consumed

Engineering Contradiction:

Improvedata processing capabilityVSAvoiddata privacy risk

Core Design Contradiction:

ProductivityVSObject-affected harmful factors

Solution Approach 1:

Instead of moving data to a centralized location for processing, the patent inverts the approach by bringing computation to where the data resides. Local processing nodes execute analytics tasks at distributed sites, eliminating the need to copy sensitive data across networks while maintaining centralized coordination of processing workflows.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent segments the centralized analytics workload into distributed processing tasks that can be executed independently at local nodes. The catalog service divides data processing into manageable units that run locally, reducing network bandwidth consumption and eliminating privacy risks associated with data centralization.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If a shared distributed file system is deployed across multiple geographic locations, then data accessibility is improved, but system complexity and maintenance difficulty increase

Engineering Contradiction:

Improvedata accessibilityVSAvoidfile system configuration and maintenance

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a catalog service as an intermediary layer between applications and distributed data storage. This catalog service manages metadata and data location information centrally, while actual data remains distributed across local file systems. This approach provides unified data accessibility without requiring a complex shared distributed file system across all locations.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent extracts the file system management complexity from the distributed architecture by separating metadata management (handled by the catalog service) from actual data storage (handled by local file systems). This extraction eliminates the need for a shared distributed file system while maintaining data accessibility across geographic locations.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If data is moved from local sites to a centralized site, then centralized analytics can be performed, but bandwidth is consumed and energy is wasted

Engineering Contradiction:

Improveanalytics processing capabilityVSAvoidbandwidth and energy consumption

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent inverts the traditional data movement paradigm by keeping data stationary and moving computation to where data resides. Local processing nodes execute analytics workflows at distributed sites, eliminating network bandwidth consumption and energy waste associated with moving large volumes of data to centralized locations.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent enables local processing nodes to autonomously execute analytics tasks using locally available data without requiring data to be transported elsewhere. Each distributed site performs its own data processing independently, reducing overall system bandwidth consumption and energy expenditure while maintaining centralized coordination of analytics workflows.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10944688B2Distributed catalog service for data processing platform

Publication Date: 2021.03.09 EMC IP HLDG CO LLC

AI summary

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The one or more processing devices are operative to configure a plurality of distributed processing nodes to communicate over a network, to abstract content locally accessible in respective data zones of respective ones of the distributed processing nodes into respective catalogs of a distributed catalog service in accordance with a layered extensible data model, and to provide in the distributed processing nodes a plurality of microservices for performing processing operations on at least one of the layered extensible data model and the catalogs. The layered extensible data model comprises a plurality of layers including a core data model layer and at least one extensions layer. The microservices may comprise at least one microservice to alter the layered extensible data model and at least one microservice to query one or more of the catalogs.