Autonomous Container Group Management for Scale Out Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems face challenges in scalable storage management and load balancing as they approach the petabyte level, with existing file systems like Veritas File System (VxFS) and XFS being inefficient in managing large disk or storage space, and experiencing decreased data write and reference update efficiency with numerous clients sending data under different policies.
Innovation Solution
The implementation of a deduplication system with autonomous data container group management, where data container groups are organized under multiple file systems, each with a reference database and a unique identifier, allowing for efficient backup processes and load balancing through a data container group location map that directs data segments based on client-policy pairs, enabling parallel data storing and reference updating.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single file system is used for storage management, then the system structure is simple, but the storage capacity management efficiency decreases and load balancing becomes problematic at petabyte level
Solution Approach 1:
The patent divides the storage system into multiple file systems, each managing a specific subset of data container groups. This segmentation allows each file system to handle smaller, more manageable portions of the total petabyte-scale storage, improving management efficiency while distributing the load across multiple systems rather than overwhelming a single file system.
2Productivity
If data container groups are distributed across multiple file systems, then load balancing improves and storage capacity management efficiency increases, but the system complexity and data organization overhead increases
Solution Approach 1:
The patent introduces a data container group location map as an intermediary structure that maps data container group identifiers to their respective file systems and subdirectories. This location map serves as a mediator that simplifies the complexity of distributing data across multiple file systems by providing a clear, centralized reference for locating and managing data container groups, thereby reducing the operational overhead of multi-file system management.
3Adaptability or versatility
If multiple clients send data under different policies to a deduplication system, then the system versatility increases, but the efficiency of data writes and reference updates decreases with large amounts of data
Solution Approach 1:
The patent segments the data container groups by assigning specific groups to specific file systems based on client-policy pairs. This segmentation allows different clients writing under different policies to be directed to appropriate, dedicated storage locations, reducing contention and improving write efficiency while maintaining the system's ability to handle diverse client requirements.
Solution Approach 2:
The patent applies local quality by creating dedicated data container groups and file system assignments tailored to specific client-policy pairs. Each client-policy combination receives customized storage allocation and management, optimizing the write and update operations for each specific client's needs rather than using a uniform approach for all clients.
Data Source
AI summary
A method for data container group management in a deduplication system is provided. The method includes arranging a plurality of data container groups according to a plurality of file systems. A subset of the plurality of data container groups correspond to each of the plurality of file systems, each of the plurality of data container groups having a reference database, a plurality of data containers, and a data container group identifier (ID). The method includes performing a first backup process for a first client-policy pair with deduplication via a first one of the plurality of data container groups and performing a second backup process for a second client-policy pair with deduplication via a second one of the plurality of data container groups.