Worldwide Distributed File System Model for Big Data Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional techniques are inadequate for analyzing and managing 'Big Data' due to its sheer volume and distribution across multiple clusters and clouds, as they fail to perform analysis across administrative boundaries and lack the necessary storage, processing capacity, and bandwidth to handle large datasets.

Innovation Solution

A method for performing file system activities across administrative boundaries between multiple file systems, enabling worldwide job management and results reception, which includes a worldwide distributed file system model that allows for Massive Parallel Processing (MPP) across multiple clusters and clouds, facilitating the analysis of Big Data sets regardless of their location.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional techniques are used for data analysis, then administrative boundaries and system limitations are maintained, but the ability to analyze Big Data across multiple clusters and clouds is insufficient

Engineering Contradiction:

Improveability to analyze Big Data across multiple clusters and cloudsVSAvoidsystem complexity for managing distributed file systems

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the distributed file system into multiple independent file system instances, each managed separately but accessible through a unified interface. This allows Big Data analysis across administrative boundaries while maintaining manageable complexity through modular organization of file system components.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary layer that provides unified access to multiple distributed file systems across different administrative boundaries. This intermediary manages the complexity of cross-system operations while enabling versatile Big Data analysis capabilities without requiring direct complex interactions between all system components.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If data is distributed across multiple administrative boundaries, then storage capacity and processing power are increased, but management and coordination become more difficult

Engineering Contradiction:

Improvestorage capacity and processing powerVSAvoidmanagement and coordination of distributed systems

Core Design Contradiction:

Quantity of substanceVSEase of operation

Solution Approach 1:

The system implements a universal interface that provides multi-functional access to distributed file systems across multiple administrative boundaries. This universal access mechanism enables increased storage capacity and processing power while simplifying management operations through a standardized, consistent interface that handles diverse underlying systems uniformly.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Productivity

If conventional file system management is used, then individual system control is maintained, but bandwidth and processing capacity for large datasets are insufficient

Engineering Contradiction:

Improvebandwidth and processing capacityVSAvoidcomplexity of distributed file system architecture

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent extends the traditional single-dimension file system architecture by adding the dimension of cross-administrative boundary access. This dimensional expansion enables increased bandwidth and processing capacity for large datasets by utilizing multiple distributed file systems simultaneously, while the systematic approach to this extension manages the inherent architectural complexity.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS9747127B1Worldwide distributed job and tasks computational model

Publication Date: 2017.08.29 EMC IP HLDG CO LLC

AI summary

Example embodiments of the present invention relate to a method, an apparatus, and a computer program product for performing file system activities across administrative boundaries between a plurality of file systems. The method includes receiving a worldwide job to perform on a plurality of file systems, managing the worldwide job, and receiving results of the worldwide job from the plurality of file systems.