Bucket-Level Change Tracking for Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in efficiently managing and replicating large numbers of storage objects due to access bottlenecks and inconsistent transaction handling across multiple storage nodes, leading to performance issues and resource underutilization.
Innovation Solution
A distributed change tracking mechanism is integrated into a distributed object storage database, utilizing a primary and secondary lookup index to track changes at a bucket level, allowing parallel processing and flexible replication, caching, and archival capabilities across peer buckets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single database stores a collection of object names, then namespace requests can be processed centrally, but access bottleneck occurs and performance deteriorates as number of objects grows
Solution Approach 1:
The patent divides the centralized database into multiple distributed databases, each responsible for tracking objects in specific buckets. This segmentation eliminates the single database bottleneck by distributing namespace request processing across multiple nodes, allowing parallel handling of object name lookups and reducing access concentration to a single point.
2Quantity of substance
If distributed storage system stores large numbers of objects, then storage capacity increases, but efficient management becomes difficult and performance issues arise
Solution Approach 1:
The patent segments the storage system into multiple distributed databases, each managing objects in specific buckets. This allows the system to scale horizontally by adding more databases rather than relying on a single database to manage all objects, maintaining management efficiency even as the total number of objects grows.
Solution Approach 2:
The patent introduces a bucket-level organization dimension, where objects are grouped into buckets and managed by different databases. This dimensional reorganization allows efficient management of large object quantities by distributing the management burden across multiple databases rather than increasing the complexity of a single database.
3Reliability
If change tracking is implemented at object level, then data protection improves, but system complexity and overhead increase
Solution Approach 1:
The patent implements change tracking at the bucket level rather than at the individual object level. Each bucket has its own database that tracks changes locally, providing data protection and consistency within each bucket while avoiding the complexity of tracking changes across the entire distributed system at the object level.
Data Source
AI summary
In one embodiment, distributed data storage systems and methods integrate a change tracking manager with scalable databases. According to one embodiment, a computer implemented method comprises integrating change tracking of storage objects into the distributed object storage database that includes a first database of a first type and one or more chapter databases of a second type with the distributed object storage database supporting a primary lookup index and a secondary lookup index in order to locate a storage object. The method includes recording in a header of a chapter database a network topology for connecting a bucket having the chapter database to a first peer bucket when a new mirror to the first peer bucket is being established, and recording a first directive into the header of the chapter database to express a type of content to be mirrored from the bucket to the first peer bucket.