Storage Virtualization Metadata Caching for Large Transaction Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage virtualization systems face limitations in managing large numbers of data objects due to finite internal memory constraints, which restrict their capacity to handle increased storage demands from larger NAS systems without affecting performance or causing data loss.
Innovation Solution
The system supplements primary internal memory with a secondary memory store, optimizing memory usage by caching metadata between primary and secondary memory stores, allowing for efficient management of significantly more data objects through the use of multiple data structures and caching algorithms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a storage virtualization system uses only primary internal memory to store metadata, then access speed and performance are maintained, but the system can only manage a limited number of data objects (50-60 million) due to finite memory capacity
Solution Approach 1:
The patent divides the memory system into two segments: primary internal memory for frequently accessed metadata and secondary external memory for less frequently accessed metadata. This segmentation allows the system to manage over 500 million data objects by storing metadata across both memory types, while maintaining performance by keeping critical metadata in fast primary memory.
Solution Approach 2:
The patent adds a spatial dimension to the memory architecture by introducing external memory storage. This transforms the system from a single-dimension (internal memory only) to a two-dimension memory system that combines internal and external storage, enabling expanded capacity while maintaining access efficiency through strategic data placement.
2Quantity of substance
If a storage virtualization system expands memory capacity to handle more data objects, then storage capacity increases, but hardware costs and system complexity increase
Solution Approach 1:
The patent makes the external memory system multi-functional by using it both as extended storage capacity and as a cost-effective alternative to expanding primary memory. The same external memory infrastructure serves dual purposes: storing large volumes of metadata and providing an economical solution that avoids the high cost of expanding internal memory while managing 500 million+ data objects.
Solution Approach 2:
The patent introduces a memory management component that acts as an intermediary between primary and secondary memory systems. This mediator handles metadata caching, retrieval, and synchronization, abstracting the complexity of the two-tier memory architecture from the core storage virtualization functions and simplifying implementation.
3Quantity of substance
If a storage virtualization system uses external memory to store metadata, then storage capacity increases significantly, but access time may increase due to slower external memory speeds
Solution Approach 1:
The patent implements preliminary action by proactively caching frequently accessed metadata in primary internal memory before it is needed. The system monitors access patterns and pre-loads metadata into fast primary memory, ensuring that commonly accessed data objects can be retrieved at high speed while less frequently accessed metadata resides in external memory.
Solution Approach 2:
The patent substitutes the mechanical limitation of primary memory capacity with an intelligent caching system that uses software-based memory management. Instead of relying solely on physical memory expansion, the system uses virtual memory techniques and caching algorithms to provide fast access to frequently used metadata while storing less frequently accessed metadata in external storage.
Data Source
AI summary
The present invention provides a system and method for handling large transactions by a storage virtualization system. Current storage virtualization systems can only manage a limited number of data objects stored on an associated NAS system because of the cost or hardware constraints of physical main memory. The present invention will enable a storage virtualization system to increase the number of data objects managed by a factor of five or more by using an associated secondary memory store, such as a disk, to store information or metadata about data objects stored on associated NAS systems. The invention also provides a method for moving metadata from the storage virtualization system memory to the secondary store and back, and recovering metadata if corrupted. The invention thereby increases the number of data objects handled by the storage virtualization system with negligible loss of performance as perceived by clients of the storage virtualization system.