Elastic Data Agent Scaling for Kubernetes Backup Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection methods for application orchestrator computing environments, such as Kubernetes clusters, are inefficient in scaling resources to meet varying data protection needs, leading to potential cost overruns and service level agreement breaches, while lacking interoperability across different computing environments.
Innovation Solution
An illustrative data storage management system deploys an elastic number of data agents based on scaling factors, including characteristics of the source data and cluster resources, using a coordinator data agent to auto-scale backup resources within and across Kubernetes clusters, and optionally utilizing an infrastructure cluster for on-demand deployment of data agents, minimizing overhead on the production cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed number of data agents are deployed for data protection, then the system structure is simple and easy to manage, but the system cannot adapt to varying workload demands leading to cost overruns and service level agreement breaches
Solution Approach 1:
The system implements dynamic scaling of data agents by deploying additional agents on-demand based on workload conditions and tearing them down when no longer needed. The coordinator data agent monitors cluster conditions and automatically adjusts the number of backup data agents, transforming the static agent deployment into a dynamic system that adapts to changing data protection requirements.
Solution Approach 2:
The coordinator data agent autonomously manages the deployment and teardown of backup data agents based on monitored cluster conditions and workload demands. The system self-adjusts without external intervention, automatically scaling resources up or down based on real-time conditions, thereby resolving the contradiction between adaptability and management complexity.
2Reliability
If data protection resources are scaled up to meet peak demands, then service level agreements are maintained, but processing loads and costs increase
Solution Approach 1:
The system dynamically adjusts the number of active data agents based on real-time workload conditions. During peak demands, additional agents are deployed to maintain service level agreements; during low-demand periods, agents are torn down to reduce processing loads and costs. This dynamic approach resolves the contradiction by making resource allocation flexible rather than static.
Solution Approach 2:
The system changes the parameter of data agent quantity based on workload conditions. By monitoring cluster health, data volume, and protection requirements, the coordinator agent adjusts the number of active data agents, transforming the fixed resource allocation into a variable parameter that optimizes both reliability and resource efficiency.
3Adaptability or versatility
If multiple clusters require data protection, then comprehensive data coverage is achieved, but the complexity of managing data agents across clusters increases
Solution Approach 1:
The coordinator data agent is designed with universal functionality to manage backup operations across multiple application orchestrator clusters. A single coordinator agent can deploy and manage backup agents in different clusters, providing multi-cluster coverage through one unified management interface, thereby reducing the complexity that would otherwise arise from managing separate agent deployments for each cluster.
Solution Approach 2:
The coordinator data agent serves as an intermediary between the storage manager and multiple production clusters. It receives instructions from the storage manager and translates them into cluster-specific backup operations, simplifying multi-cluster management by providing a single point of coordination rather than requiring direct management of agents in each cluster.
4Reliability
If data agents are deployed directly in production clusters, then data protection is immediate and effective, but overhead on the production cluster increases
Solution Approach 1:
The system extracts the backup coordination function from the production cluster by deploying a coordinator data agent that manages backup operations externally. The coordinator agent tears down backup data agents when no longer needed, removing unnecessary overhead from the production cluster while maintaining effective data protection through on-demand agent deployment.
Solution Approach 2:
The system implements dynamic deployment of backup data agents only when needed for data protection operations. During backup operations, agents are deployed to the production cluster to ensure effective protection; during non-operational periods, agents are torn down to minimize overhead. This dynamic approach balances reliability and resource consumption.
Data Source
AI summary
Data protection resources are automatically scaled to the needs of data source(s) in an application orchestrator computing environment, such as a cluster in a Kubernetes deployment. The approach is adaptable to data sources in production clusters or application suites that are not application orchestrator deployments, such as a cloud-based database-as-a-service (DBaaS). A data storage management system protects cluster-based data with an elastic number of data protection resources (e.g., data agents, media agents), which are deployed on demand. The number of data protection resources deployed for a particular job are appropriate to the workload(s) at present and depend on a variety of scaling factors. In some embodiments, data protection resources are deployed within the same cluster as the data sources. In other embodiments, a separate infrastructure cluster provides the data protection resources on demand, and connects to any number and types of data sources, whether cloud-based or otherwise, without limitation.