Application-Aware Adaptive Sharding for Backup I/O Throttling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data management systems face challenges in managing data backups due to input/output (I/O) throttling in cloud storage environments, particularly when dealing with snappables like exchange accounts that have a large number of small items, leading to increased likelihood of backup slowness and failures, and duplicative copies across shards limit deduplication performance.
Innovation Solution
The system maps snappable types to shards based on application types, determining the total number of shards needed to accommodate the highest number of computing objects per application, minimizing shards per application type, and optimizing deduplication by distributing computing objects across a limited number of shards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of shards is increased to avoid I/O throttling limits, then backup reliability is improved, but deduplication performance deteriorates due to duplicative copies across shards
Solution Approach 1:
The system segments computing objects into different shard groups based on their application types, creating a hierarchical structure where deduplication can operate effectively within each application-specific shard while maintaining multiple shards for reliability. This segmentation allows the system to achieve both goals: improved backup reliability through multiple shards and maintained deduplication performance within application-bound shards.
2Productivity
If snappables are distributed across multiple shards to prevent throttling, then backup speed is improved, but resource usage increases due to duplicative copies
Solution Approach 1:
The system applies local quality by making shards application-specific, where each shard is optimized for a particular application type. This allows deduplication to work effectively within each application's shard (reducing resource usage) while still distributing data across multiple shards for parallel backup processing (improving backup speed). The local optimization within each shard group balances both performance and resource efficiency.
3Use of energy by moving object
If application-aware sharding is implemented to optimize deduplication, then resource usage is reduced, but system complexity increases
Solution Approach 1:
The system performs preliminary action by pre-establishing mappings between application types and specific shards before backup operations begin. This upfront organization based on application types enables automatic, efficient shard selection during backup without requiring complex real-time decisions, thereby reducing resource usage through effective deduplication while keeping the system complexity manageable through pre-computed mappings.
Data Source
AI summary
Methods, systems, and devices for data management are described. The method may include determining, by a backup management system and for computing objects that are each associated with a respective application, respective quantities of computing objects associated with each application, determining a quantity of shards to use to back up the computing objects based on an upper limit and a first respective quantity of computing objects for a first application having a highest respective quantity of computing objects, mapping computing objects associated with the first application to each shard, mapping computing objects associated with the other applications to a respective subset of the shards based on the respective quantity of computing objects for the other application and the upper limit, and causing the set of computing objects to be backed up to the quantity of shards of the storage system in accordance with the mapping.