Distributed Change Tracking for Scalable Database Access Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in efficiently managing large numbers of storage objects and their names, leading to access bottlenecks and consistency issues due to centralized database approaches, which result in underutilization of resources and performance bottlenecks.
Innovation Solution
A change tracking mechanism integrated with scalable databases that allows each bucket to track relationships with peer buckets, enabling independent storage policies and efficient replication, caching, archival, and migration by using a distributed architecture with primary and secondary lookup indices and peer markers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single database stores all object names in a centralized manner, then data protection and consistency are improved, but access bottlenecks and performance degradation occur as the number of objects grows
Solution Approach 1:
The patent divides the centralized database into multiple distributed databases, each responsible for tracking objects in specific buckets. This segmentation allows the system to scale horizontally by adding more databases as the number of objects grows, eliminating the single point of congestion while maintaining data protection through distributed replication.
Solution Approach 2:
The patent introduces a hierarchical dimension to the database structure by organizing databases into groups, where each database tracks a subset of buckets. This dimensional organization enables parallel processing of namespace requests across multiple databases, improving access performance while maintaining consistency through coordinated replication.
2Stability of the object's composition
If a single database stores all object names, then data consistency is maintained, but resource utilization becomes inefficient and bottlenecks increase
Solution Approach 1:
By segmenting the database into multiple distributed instances, each handling a portion of the namespace, the system achieves better resource utilization. Each database operates independently with its own resources, reducing the bottleneck effect of a single centralized database while maintaining consistency through coordinated replication across the distributed system.
Solution Approach 2:
The patent changes the scale parameter of the database system from a single large database to multiple smaller databases. This parameter change allows the system to handle larger total object counts by distributing the workload, improving resource utilization efficiency while maintaining data consistency through replication protocols.
3Device complexity
If centralized database approach is used, then implementation simplicity is maintained, but scalability and flexibility are limited
Solution Approach 1:
The patent implements segmentation of the database into distributed instances that can be independently scaled. This allows the system to maintain implementation simplicity through standardized database components while achieving high scalability by adding more database instances as needed, supporting flexible adaptation to varying storage requirements.
Solution Approach 2:
The distributed database architecture provides multi-functionality by enabling the same database infrastructure to serve multiple purposes: namespace tracking, data protection, scalability, and flexibility. The system can adapt to different storage needs and configurations while using the same core database components, improving both scalability and implementation efficiency.
Data Source
AI summary
In one embodiment, distributed data storage systems and methods are described for integrating a change tracking manager with scalable databases. According to one embodiment, a computer implemented method comprises managing storage of objects and continuously tracking changes of the objects in a distributed object storage database, creating a record for an object having an object name, the object being stored in a bucket of the distributed object storage database, linking the bucket to a peer bucket based on a directive, generating a peer marker field for the record to store one peer marker of multiple different peer markers depending on a relationship between the bucket and the peer bucket; and automatically adding a work item for the object to the secondary index of a chapter database based on the record being created in the bucket and the peer marker for the peer bucket.