Cloud Directory Services Scalability via Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Many software applications are not designed with scalability in mind, limiting their ability to efficiently serve a large number of users and handle massive data requests, as they often lack the necessary network and computing support for dynamic data partitioning and replication.
Innovation Solution
A computer system provides cloud-scale directory services by partitioning data into multiple partitions with primary writable and secondary read-only copies, using dynamic copy locators to allocate data efficiently across partitions, and implementing single master storage systems for multi-master directory services to ensure data is promptly replicated and delivered to clients in a virtualized manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If software applications are designed with scalability in mind using data partitioning and replication, then the application can serve a large number of users efficiently, but the system complexity and infrastructure requirements increase significantly
Solution Approach 1:
The directory service data is divided into multiple partitions, each stored on separate servers. This segmentation allows the system to scale horizontally by adding more partition servers, enabling support for large numbers of users while distributing the computational and storage load across multiple independent units, thereby managing system complexity through modular organization
Solution Approach 2:
The system creates read-only copies (replicas) of data partitions across multiple servers. This copying mechanism enables parallel read operations to occur simultaneously on multiple servers, significantly improving the system's capacity to serve multiple users concurrently while maintaining data availability and reducing the load on any single server
2Speed
If data is partitioned into multiple copies with primary and secondary replicas, then data delivery efficiency to clients improves, but the complexity of managing and locating data copies increases
Solution Approach 1:
The system introduces a namespace service as an intermediary layer between clients and data partitions. This mediator maintains mappings between logical data identifiers and physical storage locations, automatically routing client requests to the appropriate primary or secondary copies. This intermediary simplifies data copy management by abstracting the complexity of replica location and selection from the core directory service logic
Solution Approach 2:
The system pre-establishes multiple secondary read-only copies of each data partition before they are needed for serving clients. This preliminary replication ensures that when client requests arrive, data can be immediately delivered from the nearest or least-loaded secondary copy, improving data delivery speed while the namespace service has already prepared the distribution map
3Reliability
If a single master storage system is used for multi-master directory services, then data consistency is maintained, but the complexity of implementing single master on multi-master store increases
Solution Approach 1:
The directory service store is segmented into multiple data partitions, each with its own master copy. This segmentation allows the system to implement a single-master model at the partition level while maintaining multi-master capability across the distributed store. Each partition's master copy ensures data consistency within that partition, while the overall system achieves scalability through the distributed partition architecture
Solution Approach 2:
The namespace service acts as an intermediary that coordinates between multiple master copies across different partitions. It maintains the authoritative mapping of data identifiers to partition locations, ensuring that clients access the correct master copy for each data element, thereby maintaining data consistency across the multi-master distributed store without requiring complex inter-master communication protocols
Data Source
AI summary
Embodiments described herein are directed to providing scalability to software applications. A computer system partitions a portion of data stored in a directory services system into multiple different data partitions. Each data partition includes a primary writable copy and at least one secondary read-only copy of the data. The computer system receives a client request for a portion of the data that is stored in the directory services system and accesses various stored partition mappings to determine which of the different data partitions includes the requested data. The computer system also accesses a dynamic copy locator to determine which of the read-only copies of the indicated partition to access and provide the accessed primary writeable copy of the indicated partition and the determined read-only copy to the client in a virtualized manner so that the client is not aware of the data partitions.