Two-Level Addressing in Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage cluster technologies face inefficiencies in managing large volumes of data, leading to increased CPU load and slower access times as the number of files and nodes grows, particularly due to the reliance on multicast techniques that interrupt nodes and require redistribution of data upon node additions or deletions.
Innovation Solution
A two-level addressing scheme utilizing a page mapping table and pages index stored in RAM across nodes to determine the location of digital objects, allowing for efficient storage and retrieval without interrupting nodes and minimizing data redistribution, with a coordinator node managing updates and replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multicast technique is used to find objects in storage cluster, then any node can store any object providing flexibility, but all nodes must be interrupted and contacted which increases CPU load and access time
Solution Approach 1:
The patent segments the cluster address space into multiple namespaces, with each namespace managed by a dedicated namespace node. This segmentation allows objects to be routed to specific namespaces based on their address, eliminating the need to contact all nodes and significantly reducing access time while maintaining flexible object storage distribution across the cluster.
Solution Approach 2:
The patent introduces namespace nodes as intermediaries between clients and data nodes. These namespace nodes maintain namespace maps that track object locations, acting as mediators that receive object address requests, determine the appropriate data node, and route requests accordingly. This intermediary layer eliminates the need for multicast to all nodes while preserving the flexible storage capability.
2Adaptability or versatility
If multicast technique is used to find objects in storage cluster, then object location flexibility is maintained, but CPU load on all nodes increases due to interrupts
Solution Approach 1:
The patent segments the cluster address space into multiple namespaces, with each namespace managed by a dedicated namespace node. This segmentation allows objects to be routed to specific namespaces based on their address, eliminating the need to contact all nodes and significantly reducing access time while maintaining flexible object storage distribution across the cluster.
Solution Approach 2:
The patent introduces namespace nodes as intermediaries between clients and data nodes. These namespace nodes maintain namespace maps that track object locations, acting as mediators that receive object address requests, determine the appropriate data node, and route requests accordingly. This intermediary layer eliminates the need for multicast to all nodes while preserving the flexible storage capability.
3Quantity of substance
If flat address space is used across multiple nodes, then storage capacity scales with cluster size, but finding objects becomes slower as cluster grows
Solution Approach 1:
The patent segments the cluster address space into multiple namespaces, each managed by a dedicated namespace node. This segmentation allows the system to scale storage capacity by adding more namespaces and namespace nodes, while object retrieval speed is maintained because the segmented structure enables direct routing to the appropriate namespace without scanning the entire cluster.
Solution Approach 2:
The patent introduces a new dimensional organization by creating a hierarchy with namespace nodes at one level and data nodes at another. This two-dimensional structure (namespace dimension × data dimension) allows the system to scale storage capacity by adding nodes in either dimension while maintaining retrieval speed through the organized routing structure.
4Adaptability or versatility
If nodes are added or removed from storage cluster, then cluster scalability is improved, but data redistribution is required which increases complexity and time
Solution Approach 1:
The patent segments the cluster address space into multiple namespaces, with each namespace independently managed by a namespace node. When nodes are added or removed, only the affected namespace needs to be updated, not the entire cluster. This segmentation isolates the impact of node changes, reducing redistribution complexity and enabling seamless scalability.
Solution Approach 2:
The patent implements dynamic namespace management where namespace nodes can be added or removed and the system automatically adapts. The namespace maps are dynamically updated to reflect current node configurations, allowing the cluster to scale flexibly without requiring complex manual data redistribution procedures.
Data Source
AI summary
Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written by storing the stream on any suitable node and then updating a pages index stored within the cluster responsible for knowing the location of digital objects having unique identifiers that fall within a particular address range. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index.