Striped Storage Lookup Distribution via Cache Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In clustered storage systems, the central directory node handles all directory operations, including lookup requests, leading to a bottleneck due to frequent lookup operations and inefficient cache management, especially when maintaining and invalidating data container attributes during creation and deletion.
Innovation Solution
Implementing a mechanism where attributes of data containers are cached in one storage system, allowing lookup operations to be directed to the same system that caches metadata, eliminating the need for relay to the central directory node and optimizing cache efficiency by ensuring that only one storage system handles attribute removal and invalidation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all directory operations including lookup requests are handled by the central directory node, then centralized control and management are maintained, but a bottleneck occurs due to frequent lookup operations
Solution Approach 1:
The patent segments the directory operation handling by distributing lookup requests to multiple cache nodes based on hash values of data container attributes. Instead of one central node handling all lookups, the system divides the workload across N cache nodes, each responsible for a portion of the lookup operations determined by hashing the attribute name modulo N.
Solution Approach 2:
The patent introduces cache nodes as intermediaries between clients and the central directory. These cache nodes store copies of data container attributes and handle lookup requests locally, acting as mediators that reduce the burden on the central directory node while maintaining consistency with the centralized metadata.
2Productivity
If data container attributes are cached across multiple storage systems, then lookup load is distributed, but cache invalidation becomes complex when maintaining consistency
Solution Approach 1:
The patent implements a feedback mechanism where the central directory node notifies cache nodes of attribute changes. When the central directory updates a data container attribute, it generates a notification that is sent to the appropriate cache node, which then updates or invalidates its cached copy, ensuring consistency without complex manual management.
Solution Approach 2:
Each cache node independently manages its own cache validity by receiving notifications from the central directory. The cache nodes autonomously update their cached attributes when notified, eliminating the need for complex coordinated invalidation protocols across the distributed system.
3Measurement precision
If lookup requests are relayed through the central directory node, then centralized metadata accuracy is ensured, but lookup operation speed decreases
Solution Approach 1:
The patent performs preliminary actions by having cache nodes pre-cache data container attributes from the central directory before lookup requests arrive. This advance preparation allows cache nodes to immediately serve lookup requests using their cached copies, significantly speeding up lookups while the central directory maintains the authoritative metadata.
Solution Approach 2:
Cache nodes serve as intermediaries that provide fast local access to cached attributes, reducing the need for clients to directly query the central directory for every lookup. The intermediaries maintain accuracy by receiving updates from the central directory while providing speed through local caching.
4Device complexity
If the central directory node services all directory operations, then simplified architecture is maintained, but performance bottleneck occurs during frequent lookups
Solution Approach 1:
The patent segments the directory operation workload by distributing lookup requests to multiple cache nodes based on hash values. The central directory node retains responsibility for metadata management and attribute updates, while cache nodes handle specific portions of lookup operations, effectively dividing the system's functional responsibilities to improve throughput.
Solution Approach 2:
The central directory node maintains multiple functions: it serves as the authoritative metadata store, handles attribute updates, and manages cache coherence by notifying cache nodes of changes. Cache nodes provide multi-functionality by storing cached attributes, handling lookup requests for their assigned portions, and participating in the distributed cache coherence mechanism.
Data Source
AI summary
The present invention provides a mechanism for storing and accessing attributes of a data container (e.g., characteristics such as a name of a data container used to perform a lookup operation against the data container) in cache memory within storage systems in a cluster of storage systems. Further, a mechanism is provided so that an attribute of a data container is inserted into a cache memory of preferably one storage system (in the cluster) that caches metadata of the data container (e.g., information describing the data container). As a result, a lookup operation for a data container is directed to and served by the storage system that caches the metadata and the attribute of the data container. The lookup request is not relayed to a storage system that maintains a central directory of attributes of all data containers in the cluster.