Data Movement Task Distribution in Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing clusters face inefficiencies in distributing and managing large data movement tasks across multiple nodes, leading to contention issues and reduced performance due to cross-node storage communication and lock negotiation, especially when handling billions of files or directories.
Innovation Solution
The solution involves assigning different affinities to node groups for processing data movement tasks across storage segments, using sharding to partition the data, and optimizing start positions within these segments to minimize contention, as well as generating tasks in claimed or unclaimed states to reduce overhead and improve task distribution efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data movement tasks are distributed across multiple nodes in a computing cluster, then the data movement capacity and fault tolerance are improved, but cross-node storage communication and lock negotiation increase, leading to contention issues and reduced performance
Solution Approach 1:
The patent applies segmentation by dividing the storage system into distinct namespaces, where each namespace is assigned to specific nodes. This segmentation isolates storage communications within namespaces, reducing cross-node communication overhead and lock negotiation complexity while maintaining distributed data movement capacity across multiple nodes.
2Adaptability or versatility
If multiple nodes share access to the same storage segments, then resource utilization is improved, but contention and lock negotiation overhead increase
Solution Approach 1:
The patent segments storage resources into namespace-specific partitions, allowing nodes to access shared storage resources within their assigned namespace without competing for locks on the same segments. This reduces lock negotiation time while maintaining high resource utilization through controlled sharing within namespaces.
Solution Approach 2:
The patent implements local quality by assigning specific nodes as primary owners of particular namespaces, giving those nodes preferential access and reducing lock negotiation overhead. Other nodes can still access the storage resources but with lower priority, thereby reducing contention and lock negotiation time while maintaining overall resource utilization.
3Ease of operation
If a centralized task distribution mechanism is used, then task management simplicity is improved, but single points of failure and communication bottlenecks increase
Solution Approach 1:
The patent segments the task distribution mechanism into namespace-specific task queues managed by individual namespace owners rather than a single centralized queue. Each namespace owner independently manages task distribution for their assigned namespaces, eliminating single points of failure while maintaining operational simplicity through localized task management.
Solution Approach 2:
The patent transitions from a single-dimensional centralized task queue to a multi-dimensional distributed task management system where task queues are organized by namespace. This dimensional change allows parallel task distribution across multiple namespace owners, improving fault tolerance while maintaining task management simplicity through consistent namespace-based organization.
4Adaptability or versatility
If nodes frequently communicate with shared storage for task claims, then task distribution flexibility is improved, but communication overhead and contention increase
Solution Approach 1:
The patent segments storage access patterns by namespace, allowing nodes to efficiently claim tasks within their assigned namespaces without frequent cross-node storage communications. This segmentation reduces communication overhead and energy consumption while maintaining task distribution flexibility through namespace-based task queues and selective access patterns.
Data Source
AI summary
The described technology is generally directed towards data movement from data storage clusters. A data movement job can be split into data movement tasks, and the data movement tasks can be distributed among multiple nodes of a data storage cluster. In order to distribute the tasks, a data store that identifies the tasks can be split into multiple segments, and different groups of nodes can be assigned affinities to process tasks identified in the different segments. Furthermore, the nodes of each group of nodes can be assigned different start positions within their storage segment, to reduce the likelihood of contention between nodes in connection with claiming a same task. Some of the tasks can be generated in an already claimed state, which also reduces contention and allows for more efficient operation.