Distributed File System Notification Mechanism

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Scalable data storage systems face challenges in efficiently notifying applications of changes to files and new file additions, leading to increased latency and resource consumption due to continuous scanning processes, which can cause performance degradation and failure in handling large volumes of digital data.

Innovation Solution

A distributed file system with a notification system that communicates with applications, allowing them to register interest in specific file events, thereby eliminating the need for continuous scanning and reducing latency by immediately notifying them of changes, such as new file additions, deletions, or updates.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a scanning tool continuously scans the storage system to detect new material, then the application can detect file changes, but network bandwidth and processing bandwidth are consumed and latency is introduced

Engineering Contradiction:

Improvedetection of new materialVSAvoidlatency from storage to detection

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Instead of the application continuously scanning the storage system to detect changes (pull model), the storage system actively notifies the application of changes (push model). This inversion of the detection approach eliminates the need for continuous scanning while ensuring immediate detection of file changes.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The storage system implements a feedback mechanism where it monitors its own state for changes (new files, deletions, modifications) and automatically sends notifications to registered applications. This feedback loop ensures applications are immediately informed of changes without needing to poll or scan continuously.

Inventive Principle:

Principle #23Feedback

2Reliability

If a scanning tool continuously scans the storage system to detect new material, then the application can detect file changes, but network bandwidth and processing bandwidth are consumed

Engineering Contradiction:

Improvedetection of new materialVSAvoidnetwork and processing bandwidth

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The storage system performs self-monitoring of its own state changes and automatically generates notifications when changes occur. This self-service approach eliminates the need for external scanning tools to consume network and processing bandwidth, as the storage system itself detects and reports changes.

Inventive Principle:

Principle #25Self-service

3Quantity of substance

If hundreds of disk drives are networked together to increase storage capacity, then storage capacity is improved, but the chances of failure increase

Engineering Contradiction:

Improvestorage capacityVSAvoidsystem failure risk

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The storage system is divided into multiple independent disk drives that can be individually monitored and managed. Each drive operates as a separate unit within the distributed network, allowing failures to be isolated to individual components rather than affecting the entire system.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The notification system implements redundancy by maintaining registration information for multiple applications and having backup notification mechanisms ready. When failures occur in the distributed system, the notification infrastructure is designed to cushion the impact by ensuring critical notifications are delivered through available paths.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS7941455B2Notification for a distributed file system

Publication Date: 2011.05.10 HARMONIC INC

AI summary

A method for notifying an application coupled to a distributed file system is described. A command for a file for a distributed file system is received. The distributed file system stores portions of files across a plurality of distinct physical storage locations. The command for the file is compared with a notification table of the distributed file system of the distributed file system. At least one application communicates with the distributed file system. The notification system notifies the corresponding application associated with the command with the notification system.