Kubernetes File Client Backup Without API Server Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection solutions for Kubernetes clusters face challenges such as overwhelming the API server during large data transfers, which can be interrupted by load balancer switches, leading to incomplete backups and resource inefficiencies.
Innovation Solution
A scalable and resilient data backup approach that circumvents the API server by using a sponsor data agent to deploy a Kubernetes File Client or File System Data Agent within the cluster, transferring data directly to an off-cluster media agent, and optimizing network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the API server is used as a data transfer node for backup operations, then data transfer can be initiated through the existing cluster interface, but the API server becomes overwhelmed and data connections are interrupted
Solution Approach 1:
The patent extracts the data transfer function from the API server by deploying dedicated data agents within Kubernetes pods. These agents handle all data transfer operations directly, separating this function from the API server which only manages control plane communications. This extraction resolves the contradiction by allowing easy data transfer initiation through specialized components while preventing API server overload and connection interruptions.
Solution Approach 2:
The patent introduces data agents as intermediary components between the Kubernetes cluster and external backup systems. These agents reside within pods and handle data transfer operations, acting as mediators that prevent direct communication between the API server and external systems. This intermediary approach maintains operational ease while ensuring API server stability and connection reliability.
2Productivity
If the API server is used for large data transfers, then existing cluster resources are utilized, but performance deteriorates due to design limitations
Solution Approach 1:
The patent segments the backup functionality into control plane operations (handled by the API server) and data plane operations (handled by dedicated data agents). This segmentation allows the API server to focus on lightweight control tasks while data agents handle large-volume data transfers. The result is improved overall productivity through specialized resource allocation and significantly enhanced data transfer speeds without API server bottlenecks.
3Ease of operation
If load balancer switches resources during backup, then load balancing is maintained, but backup jobs are interrupted and must restart
Solution Approach 1:
The patent implements self-service capabilities within data agents that allow them to dynamically adapt to load balancer resource switching. The agents can re-establish connections and resume operations automatically when resource switches occur, maintaining backup job continuity without requiring manual intervention or causing job failures. This self-service approach preserves load balancing benefits while ensuring uninterrupted backup operations.
Data Source
AI summary
Because Kubernetes clusters can be ephemeral, backing up in-cluster data to storage outside the cluster is important. Prior art solutions used the cluster's API server, which facilitates communications with the cluster control plane, to transfer backup data through the API server. However, the API server as a data transfer node has resiliency weaknesses and can slow down backup job performance. The present solution provides a more streamlined and scalable approach, which circumvents the API server and additionally includes more robust error checking, log capture, and realtime job monitoring to provide improved data protection resilience. The disclosed approach employs a “sponsor” data agent outside the cluster and temporarily deploys a specialized backup resource within the cluster during a backup job, such as an enhanced File System Data Agent and/or a lightweight Kubernetes File Client, both of which present substantial performance and resiliency advantages over the API server.