Peer Cache Requests in Distributed Filesystem Cloud Controllers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing network storage systems face challenges in maintaining data consistency and scalability, particularly in cloud-based storage solutions, where access latency and network failures can lead to data consistency issues and increased management overhead.

Innovation Solution

A distributed filesystem system that uses multiple cloud controllers to cache and manage data, generating incremental snapshots and metadata to ensure data consistency across cloud storage systems, and employs peer cache requests to optimize data access by leveraging cached data from peer controllers, thereby reducing reliance on cloud storage systems and improving access latency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored in cloud-based storage systems, then storage capacity and scalability are improved, but access latency increases

Engineering Contradiction:

Improvestorage capacityVSAvoidaccess latency

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The system segments data into two parts: metadata (stored locally at cloud controllers) and data blocks (stored remotely at cloud storage systems). This segmentation allows the system to maintain large storage capacity while reducing access latency by keeping frequently accessed metadata locally available at cloud controllers.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Cloud controllers act as intermediaries between clients and cloud storage systems. They cache metadata and serve as local proxies, reducing the need for direct remote access to cloud storage systems and thereby decreasing access latency while maintaining scalability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If multiple cloud controllers manage distributed filesystem data, then data consistency and reliability are improved, but system complexity increases

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

Multiple cloud controllers are merged into a coordinated system that shares the burden of data consistency management. They collectively maintain metadata caches and coordinate peer cache requests, achieving improved reliability through distributed cooperation rather than single-point complexity.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

Cloud controllers create and maintain copies of metadata in their local caches. These copies enable parallel data consistency management across multiple controllers, allowing the system to achieve high reliability through redundant metadata availability while distributing complexity across multiple simpler nodes.

Inventive Principle:

Principle #26Copying

3Speed

If cloud controllers cache data locally, then access speed is improved, but data consistency challenges increase

Engineering Contradiction:

Improveaccess speedVSAvoiddata consistency

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system implements feedback mechanisms where cloud controllers receive peer cache requests for data blocks and respond with cached copies when available. This feedback loop enables fast local access while maintaining consistency through coordinated caching across multiple controllers that continuously update their metadata.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

Cloud controllers perform preliminary actions by pre-caching metadata and preparing peer cache requests before data access is needed. This allows the system to maintain fast access speeds while ensuring data consistency is established in advance through coordinated caching operations.

Inventive Principle:

Principle #10Preliminary action

4Reliability

If incremental snapshots are generated and transmitted, then data consistency is improved, but network bandwidth consumption increases

Engineering Contradiction:

Improvedata consistencyVSAvoidnetwork bandwidth

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system extracts only the essential metadata information into incremental snapshots that are transmitted between cloud controllers. By taking out only the necessary metadata rather than transferring complete data sets, the system maintains data consistency while minimizing network bandwidth consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system performs partial actions by transmitting only the incremental differences between snapshots rather than complete data sets. This partial transmission approach maintains sufficient data consistency information while significantly reducing network bandwidth requirements compared to full data synchronization.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS8805968B2Accessing cached data from a peer cloud controller in a distributed filesystem

Publication Date: 2014.08.12 PANZURA LLC

AI summary

The disclosed embodiments provide a system that archives data for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller receives a request from a client for a data block of a file stored in the distributed filesystem. Upon determining that the requested data block is not currently cached in the cloud controller, the cloud controller sends a peer cache request for the requested data block to a peer cloud controller in the distributed filesystem.