Metadata Memory Table Redistribution for Data Storage Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data storage systems face uneven workload distribution among cluster nodes, leading to inefficiencies where one node is heavily used while another experiences low workload, resulting in reduced system throughput.
Innovation Solution
Implementing a method to redistribute metadata memory tables between nodes by switching primary and backup memory tables, allowing a node with high workload to maintain a backup memory table and a node with low workload to maintain a primary memory table, thereby balancing the workload across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If metadata memory tables are concentrated on fewer nodes to improve data access efficiency, then data access speed is improved, but workload distribution becomes uneven
Solution Approach 1:
The patent implements dynamic workload balancing by allowing nodes to switch between primary and backup memory tables based on current system conditions. Nodes that become overloaded can transfer their primary memory tables to other nodes, while nodes with available capacity can take on primary responsibilities. This dynamic redistribution ensures that workload adapts to changing conditions, preventing any single node from becoming a bottleneck while maintaining fast data access through localized memory tables.
2Productivity
If workload is evenly distributed across all nodes to improve system throughput, then system throughput is improved, but data access efficiency decreases
Solution Approach 1:
The patent segments the metadata storage responsibility by dividing memory tables into primary and backup roles across different nodes. Each node maintains a primary memory table for its owned partitions and can also maintain backup memory tables for partitions owned by other nodes. This segmentation allows the system to achieve both goals: data access efficiency is maintained because each partition has a dedicated primary node, while system throughput improves because backup nodes can take over if primary nodes become overloaded.
3Reliability
If a node maintains both primary and backup memory tables to improve data availability, then data availability is improved, but resource usage increases
Solution Approach 1:
The patent changes the operational state of memory tables based on workload conditions. Nodes can dynamically switch between maintaining only primary memory tables and maintaining both primary and backup memory tables. When a node becomes overloaded, it can transfer its primary memory table to a backup node, effectively changing from a high-resource state to a lower-resource state. This parameter change allows the system to maintain data availability through backup nodes without permanently increasing resource usage on all nodes.
Data Source
AI summary
The technology described herein is directed towards balancing workload between cluster nodes via redistribution of metadata data structures (e.g., memory tables corresponding to directory table partitions). Workload-related information of a node and its partitions' primary memory tables usage is measured, and if sufficiently high, causes a move of a highly-accessed memory table (corresponding to high workload on a first node) from the first node to a second node that has less workload. The second node can contain a backup (e.g., shallow) memory table to the primary node, whereby the move can be a logical move that transforms the backup memory table into a new instance of the primary memory table on the second node. The first node's primary memory table can be deflated into a backup table on the first node that backs up the new instance of the primary table on the second node.