Peer-to-Peer Big Data Architecture for Scalable Distributed Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional distributed computing systems face scalability and robustness issues due to centralized architectures, which lead to performance bottlenecks, increased administrative costs, and single points of failure, making them inefficient for managing large datasets and big data analysis.

Innovation Solution

A peer-to-peer (P2P) network architecture that distributes data and jobs randomly across multiple nodes, using a distributed file system and task scheduler to optimize load distribution and redundancy, eliminating the need for centralized management and enabling rapid failover in case of node failures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If a master/slave architecture is used to coordinate resource allocation, then centralized control is achieved, but performance bottlenecks and reliability risks increase due to the burden on master nodes

Engineering Contradiction:

Improvecentralized controlVSAvoidreliability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent segments the centralized master node functions into distributed peer nodes. Each peer node independently manages its own resources and coordinates with other peers through a decentralized consensus mechanism, eliminating the single point of failure inherent in master/slave architectures while maintaining coordinated resource allocation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of having master nodes control slave nodes in a hierarchical structure, the patent inverts the relationship by giving all nodes equal status and decision-making power. Peer nodes collectively make resource allocation decisions through distributed consensus, transforming the control flow from top-down to bottom-up.

Inventive Principle:

Principle #13The other way round (Inversion)

2Ease of operation

If master nodes are used to manage data storage and access, then centralized management is achieved, but service disruption occurs when master nodes fail

Engineering Contradiction:

Improvecentralized managementVSAvoidservice availability

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent merges the management functions previously isolated to master nodes with all peer nodes. Each peer node maintains local data copies and can independently serve requests, combining centralized management capabilities across multiple distributed nodes to eliminate service disruption from single node failures.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent implements data replication across multiple peer nodes. Critical data and management information are copied to multiple nodes throughout the network, ensuring that if one node fails, others can continue to provide services without interruption.

Inventive Principle:

Principle #26Copying

3Quantity of substance

If hierarchical architecture is used to distribute data, then data distribution is achieved, but resource distribution becomes less distributed at higher hierarchical levels

Engineering Contradiction:

Improvedata distributionVSAvoidresource distribution

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent applies local quality by allowing each peer node to independently manage and distribute its local resources based on local conditions and needs. Rather than imposing a uniform hierarchical distribution pattern, each node adapts its resource distribution strategy to its specific context, achieving both data and resource distribution flexibility.

Inventive Principle:

Principle #3Local quality

4Ease of operation

If master/slave architecture is used for coordination, then resource allocation is centralized, but administrative costs and hardware expenses increase

Engineering Contradiction:

Improveresource allocationVSAvoidadministrative cost

Core Design Contradiction:

Ease of operationVSEase of manufacture

Solution Approach 1:

The patent implements self-service by enabling peer nodes to autonomously make resource allocation decisions based on local conditions and peer feedback. Nodes independently manage their own resource scheduling and coordination without requiring expensive centralized master node infrastructure, reducing both hardware and administrative costs.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10291696B2Peer-to-peer architecture for processing big data

Publication Date: 2019.05.14 THE ARIZONA BOARD OF REGENTS ON BEHALF OF THE UNIV OF ARIZONA

AI summary

A system is disclosed for managing large datasets. The system comprises a physical network. The physical network comprises a plurality of computing devices with a plurality of processors. The system further comprises a logical peer-to-peer (P2P) network with a plurality of nodes. The system further comprises a distributed file system for distributing data and jobs received by the system randomly across the plurality of nodes in the P2P network. The system duplicates the data to neighboring nodes of the plurality of nodes. The nodes monitor each other to reduce loss of data. The system further comprises a task scheduler. The task scheduler balances load across the plurality of nodes as tasks, derived from jobs, are distributed to various nodes. The task scheduler redistributes and forwards tasks to ensure the nodes processing the tasks are best suited to process those tasks.