Metadata Microservices Orchestration via Consistent Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems struggle to scale resources based on demand, maintain fast response times, and ensure persistence across node crashes while keeping costs low, especially in large distributed environments.
Innovation Solution
A deduplicated microservices-based storage system with a metadata microservices subsystem that uses Kubernetes for orchestration, dynamically scaling worker nodes and repartitioning metadata space using consistent hashing, combined with an in-memory read/write cache and write-ahead log for efficient deduplication and crash recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing databases are used for metadata storage and processing, then capacity can be scaled, but the system cannot adapt to changing demand conditions and lacks flexibility
Solution Approach 1:
The patent implements dynamic scaling of metadata worker nodes based on demand conditions. The system can add or remove worker nodes dynamically, and the metadata space is automatically repartitioned across the available nodes using consistent hashing. This dynamic architecture allows the system to adapt to changing demand without manual intervention while maintaining operational simplicity through automated orchestration.
2Speed
If resources are scaled up to handle peak demand, then response time improves, but cost increases
Solution Approach 1:
The system dynamically adjusts the number of metadata worker nodes based on actual demand conditions. During peak demand periods, additional worker nodes are added to maintain fast response times. During low-demand periods, nodes are removed to reduce costs. The consistent hashing repartitioning ensures seamless load distribution across varying node counts, enabling the system to optimize the trade-off between performance and cost in real-time.
3Productivity
If metadata is stored in distributed locations, then scalability improves, but maintaining persistent state across node crashes becomes more difficult
Solution Approach 1:
The system implements a master node that continuously monitors the state of worker nodes and maintains authoritative metadata about the partitioning scheme and node assignments. This central coordination point provides feedback to the distributed workers about their current state and any changes in the system configuration. The master node enables crash recovery by providing a source of truth for reconstructing the distributed metadata state, thus maintaining reliability while preserving scalability.
Solution Approach 2:
The master node acts as an intermediary between the distributed worker nodes and the external world. It manages the complexity of maintaining persistent state across crashes by centralizing the coordination logic for partitioning and node assignment. When nodes fail or are added, the master node mediates the repartitioning process using consistent hashing, ensuring that the distributed system maintains its state correctly without requiring complex peer-to-peer coordination among workers.
4Productivity
If more worker nodes are added to handle increased load, then processing capacity improves, but coordination and orchestration complexity increases
Solution Approach 1:
The master node serves as a simplifying intermediary that handles all coordination complexity. When worker nodes are added or removed, the master node automatically repartitions the metadata space using consistent hashing and updates the routing tables. This central coordination mechanism allows worker nodes to remain simple and stateless, focusing only on processing requests for their assigned partitions. The orchestration complexity is isolated to the master node, which can be easily managed and replaced without affecting the workers.
Data Source
AI summary
A deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage. A custom metric compares demand on system resources to thresholds and dynamically scales resources to add or eliminate resources according to demand. The master node partitions the range of values identifying data similarity groups across the worker nodes, reassigns ranges of values using consistent hashing as worker nodes are scaled, and maintains a mapping of ranges of values to worker IP addresses.