Elastic Data Protection Scaling for Kubernetes Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection methods for application orchestrator computing environments, such as Kubernetes clusters, are inefficient in scaling resources to meet varying data needs, leading to excessive costs and potential breaches in data security.
Innovation Solution
An elastic data protection system that deploys on-demand data agents based on workload analysis, using a coordinator data agent to manage resource scaling and distribute tasks efficiently, minimizing overhead on the storage manager and ensuring data protection across cloud and non-cloud environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data protection resources are statically allocated in application orchestrator environments, then system simplicity is maintained, but resource efficiency deteriorates leading to excessive costs
Solution Approach 1:
The patent implements dynamic resource allocation where data protection resources (data agents, media agents, storage capacity) automatically scale based on real-time workload conditions. The system monitors data generation rates, backup demands, and resource utilization metrics, then dynamically provisions or de-provisions protection resources accordingly. This resolves the contradiction by maintaining simplicity through automation while achieving resource efficiency through adaptive scaling.
Solution Approach 2:
The data protection system performs self-service by automatically detecting workload changes and adjusting its own resource allocation without external intervention. The system monitors its own performance metrics, identifies when additional protection resources are needed, and autonomously provisions them. This eliminates the need for complex manual configuration while ensuring resources are efficiently matched to actual demands.
2Loss of energy
If data protection resources are dynamically scaled to meet varying data needs, then resource efficiency is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary workload analysis component that sits between the storage manager and the data protection resources. This intermediary analyzes workload conditions, determines resource requirements, and translates them into provisioning decisions. By inserting this intelligent intermediary layer, the system achieves dynamic resource scaling without requiring complex direct control mechanisms between all system components, thus managing complexity while maintaining efficiency.
Solution Approach 2:
The system segments the data protection functionality into independent, modular components (data agents, media agents, storage managers) that can be individually scaled and managed. Each component operates semi-autonomously with well-defined interfaces, allowing the system to dynamically adjust the number and capacity of individual segments based on workload needs without redesigning the entire system architecture. This modular segmentation enables efficient resource scaling while keeping individual component complexity manageable.
3Reliability
If more data agents are deployed to handle increased workloads, then data protection capability is improved, but processing load on storage manager increases
Solution Approach 1:
The patent extracts workload analysis and resource provisioning decisions from the storage manager and places them in dedicated data agents and media agents. These extracted components independently monitor local workload conditions and make autonomous decisions about data protection operations. This extraction reduces the processing burden on the storage manager while maintaining or improving data protection capability, as the distributed agents handle the analytical and decision-making workload locally.
Solution Approach 2:
Data agents perform preliminary workload analysis and resource requirement determination before engaging the storage manager. By pre-analyzing backup demands, data generation rates, and resource availability at the agent level, the system prepares protection plans in advance, reducing the real-time processing load on the storage manager. This preliminary action allows the storage manager to execute pre-planned operations with minimal computational overhead.
Data Source
AI summary
Data protection resources are automatically scaled to the needs of data source(s) in an application orchestrator computing environment, such as a cluster in a Kubernetes deployment. The approach is adaptable to data sources in production clusters or application suites that are not application orchestrator deployments, such as a cloud-based database-as-a-service (DBaaS). A data storage management system protects cluster-based data with an elastic number of data protection resources (e.g., data agents, media agents), which are deployed on demand. The number of data protection resources deployed for a particular job are appropriate to the workload(s) at present and depend on a variety of scaling factors. In some embodiments, data protection resources are deployed within the same cluster as the data sources. In other embodiments, a separate infrastructure cluster provides the data protection resources on demand, and connects to any number and types of data sources, whether cloud-based or otherwise, without limitation.