Organization-Level Global Data Object Management via Leader-Based Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in managing organization-level global data objects, particularly in ensuring accessibility and resilience across multiple deployments, leading to unavailability and modification issues when the source deployment is down or unavailable.
Innovation Solution
Implementing a leader-based model for managing organization-level global data objects, where a single source of truth deployment acts as the leader, and all other participant deployments maintain a full copy of the data object, ensuring accessibility and resilience through a global control plane not tied to any specific account.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If a single source of truth deployment is used to manage organization-level global data objects, then data consistency and control are improved, but system availability and resilience deteriorate when the source deployment fails
Solution Approach 1:
Each participant deployment maintains a full copy of the organization-level global data object locally, enabling continued data access and operations even when the source of truth deployment is unavailable. This copying approach resolves the contradiction by providing data consistency through the source of truth while ensuring availability through local copies.
Solution Approach 2:
A global control plane acts as an intermediary that coordinates changes across all deployments. When a change is made at the source of truth, the global control plane propagates it to all participant deployments, ensuring consistency without requiring direct dependency on the source deployment for access.
2Reliability
If a leader-based model with single source of truth is implemented, then data modification control is improved, but access latency increases when the source deployment is distant or congested
Solution Approach 1:
Each participant deployment maintains a local full copy of the data object, enabling fast local read access without requiring network trips to the source of truth deployment. This local quality approach reduces access latency while the source of truth maintains ultimate modification control.
Solution Approach 2:
Data is replicated to all participant deployments in advance before any access operations occur. This preliminary action ensures that when data needs to be accessed, it is already available locally at each deployment, minimizing latency.
3Adaptability or versatility
If organization-level global data objects are made accessible from any deployment, then system versatility and flexibility are improved, but data consistency management complexity increases
Solution Approach 1:
The global control plane serves as a mediator that automatically manages change propagation across all deployments. This intermediary abstracts away the complexity of consistency management from individual deployments, allowing access from any location while maintaining consistency through automated propagation.
Solution Approach 2:
The system implements feedback mechanisms where changes made at any deployment are communicated back to the source of truth and propagated to all other deployments. This feedback loop ensures consistency is maintained automatically as data flows through the system.
Data Source
AI summary
Provided herein are systems and methods for global data objects on a data platform where the global data objects are accessible at an organization level. In particular, an organization-level global data object provided by various embodiments can be used as a generic organization object that is owned by a specific organization, and can be managed (e.g., created, deleted, or modified) by use of a leader-based model.