Versioned Database Architecture for Isolated Scenario Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently manage isolated speculative data changes, algorithm simulations, and data integrity in a scalable manner, particularly in environments with multiple users and isolated data sandboxes, while ensuring cost-effectiveness and data visibility control.
Innovation Solution
A scalable versioned database system that organizes data into scenarios and hives, allowing data visibility control, parallel execution of queries and algorithms, and supports ACID transactions through metadata databases, with data objects not modified in place and metadata objects updated transactionally, while cleaning up obsolete data by identifying unreachable data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data objects are modified in place to enable efficient updates, then write performance is improved, but data integrity and isolation between scenarios deteriorate
Solution Approach 1:
The system segments data into immutable data objects and separate metadata objects. Data objects are never modified in place but are instead replaced with new versions, while metadata objects track references to these data objects. This segmentation allows efficient updates through object replacement while maintaining data integrity through immutable data and transactional metadata updates.
Solution Approach 2:
The system creates new data objects before replacing old ones, preparing the new version in advance. This preliminary creation of new objects allows the replacement to happen atomically at the metadata level, ensuring data integrity is maintained during the update process while still achieving efficient write performance.
2Adaptability or versatility
If multiple users access data simultaneously in isolated sandboxes, then system scalability is improved, but data visibility control and isolation deteriorate
Solution Approach 1:
The system segments data access through scenario-based isolation. Each scenario has its own view of data, and the metadata database tracks which data objects are visible to which scenarios. This allows multiple users to access data simultaneously in isolated sandboxes while maintaining strict data visibility control and isolation through metadata management.
Solution Approach 2:
The metadata database acts as an intermediary between multiple users and data objects. It mediates access by tracking references, versions, and visibility rules, enabling scalable multi-user access while maintaining data isolation through transactional operations that ensure consistent views for each scenario.
3Adaptability or versatility
If speculative data changes are allowed to improve algorithm execution flexibility, then computational versatility is improved, but data cleanliness and integrity deteriorate
Solution Approach 1:
The system segments data into immutable data objects that represent the source of truth and metadata objects that handle versioning and references. This allows speculative changes to be made in isolated scenarios without affecting the cleanliness of the original data, as changes are captured as new versions rather than modifications to existing objects.
Solution Approach 2:
The system allows speculative changes to be made and then discarded when not needed, while recovering the previous clean state through versioning. The metadata database tracks all versions and references, enabling the system to discard speculative changes in isolated scenarios while maintaining data cleanliness in production environments.
4Reliability
If garbage data is not cleaned up to maintain data accessibility, then data availability is improved, but storage efficiency and system cost deteriorate
Solution Approach 1:
The system implements a feedback mechanism through reference counting in the metadata database. When all references to a data object are removed, the system receives feedback that the object is no longer needed and automatically cleans it up. This maintains data availability for active scenarios while improving storage efficiency by removing garbage data.
Solution Approach 2:
The metadata database performs self-service garbage collection by tracking references automatically. It monitors which data objects are still referenced and cleans up unreferenced objects without external intervention, maintaining data availability for active scenarios while improving storage efficiency through automatic garbage removal.
Data Source
AI summary
Systems and methods that for a scalable versioned database that can organize data into scenarios and hives. These define how data visibility is controlled by scenarios, and how data can be transferred between scenarios. In some embodiments, queries and algorithms can be executed in independent processes, which may execute in parallel, and on independent machines. Furthermore, data objects can be placed in shared storage, and metadata objects can be placed in a metadata database which supports ACID transactions. Data objects are not modified in place after they are constructed, while metadata objects can be updated transactionally, using the metadata database. Data can be updated by creating new data objects and connecting them to scenarios with new metadata objects. Finally, obsolete data may be cleaned up by identifying unreachable data, which is safe to delete.