Deduplicating SMB Change Notify Events Using Generation Numbers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed file systems, the frequent generation of change notify events can overflow buffers, leading to wastage of storage, compute, and network resources, and potentially cause process failures due to redundant event notifications.
Innovation Solution
A method using a hash table to store event entries with a generation number, where events with matching logical inode numbers and a generation number difference below a threshold are deduplicated, and user space filter buffers are used to notify clients only of unique events, thereby reducing redundant notifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a distributed file system processes all kernel write events as separate SMB change notify events, then complete file modification tracking is achieved, but buffer overflow and resource waste occur due to redundant notifications for single large writes
Solution Approach 1:
The patent merges multiple kernel write events that belong to the same logical file operation into a single SMB change notify event. By tracking the logical inode number and using generation numbers to identify related writes, the system combines redundant notifications into one consolidated event, eliminating resource waste while preserving complete modification tracking.
Solution Approach 2:
The patent introduces an intermediary deduplication layer between the kernel event buffer and the SMB change notify processing. This intermediary uses hash tables and generation numbers to filter and consolidate events before they are sent to clients, acting as a mediator that prevents buffer overflow while maintaining accurate file modification tracking.
2Loss of information
If the file system generates detailed change notify events for each kernel write, then precise file modification information is provided to clients, but the event buffer overflows due to excessive event volume
Solution Approach 1:
The patent applies partial action by selectively generating SMB change notify events only for unique logical file operations rather than for every kernel write event. By using generation numbers and hash tables to identify which events are new and which are duplicates, the system performs partial notification, reducing buffer volume while preserving essential file modification information.
Solution Approach 2:
The patent changes the parameter representation by introducing generation numbers that encode temporal and logical relationships between writes. Instead of treating each kernel write as a separate event, the system uses generation number comparison to determine event uniqueness, transforming the event processing from quantity-based to parameter-based filtering.
3Reliability
If multiple SMB change notify events are sent for a single large write operation, then complete notification coverage is achieved, but process failure may occur due to buffer flooding
Solution Approach 1:
The patent performs preliminary deduplication action by checking the hash table and generation numbers before generating SMB change notify events. This preliminary filtering prevents redundant events from being created in the first place, eliminating the risk of buffer flooding before it can occur and ensuring process stability while maintaining complete notification coverage for unique events.
Solution Approach 2:
The patent applies preliminary anti-action by pre-identifying and eliminating duplicate events using generation number comparison and hash table lookup. This preemptive measure counteracts the potential harmful effect of buffer flooding by removing redundant events before they can contribute to overflow conditions, protecting the system from process failure.
Data Source
AI summary
Implementations are provided herein for using relative generation numbers for deduplicating kernel events modification events. The process can identify multiple modification events that take place on the same inode within a configurable relative amount of time and deduplicate the events against one another. A hash table can be used to store a global list of events associated with inodes, and thus only the hash table need be deduplicated. Filter buffer(s) setup when an Server Message Block (“SMB”) client requests a change notifications on a file and/or directory can then use the data from the hash table(s) to notify clients of change notify events.