Decoupled Database Architecture for Low-Latency Edits and Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face a tradeoff between low-latency complex querying and low-latency editing, with OLAP systems prioritizing querying over editing and OLTP systems prioritizing editing over querying, making it difficult to achieve both simultaneously, especially with large datasets.
Innovation Solution
A database system with a decoupled compute and storage layer, comprising a canonical dataset for historical data, a durable buffer for recent edits, and an edits dataset for accumulated edits, utilizing a query rewriter to synthesize a real-time view by joining these datasets, enabling low-latency atomic edits and reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If OLAP database systems are designed for complex analytical queries with rapid execution time, then query performance is improved, but write performance and update latency deteriorate
Solution Approach 1:
The patent segments the database system into distinct components: a canonical dataset for historical data, a durable buffer for recent edits, and an edits dataset for accumulated edits. This segmentation allows each component to be optimized for its specific function, enabling both fast queries and fast updates simultaneously.
Solution Approach 2:
The patent introduces a query rewriter as an intermediary component that synthesizes real-time views by joining data from the canonical dataset, durable buffer, and edits dataset. This intermediary enables the system to provide both OLAP-style analytical querying and OLTP-style low-latency updates without requiring a choice between the two paradigms.
2Speed
If versioned datasets are used for reducing query latency by storing multiple data copies, then query performance is improved, but system complexity and data storage requirements increase tremendously
Solution Approach 1:
Instead of creating multiple complete versioned copies of the entire dataset, the patent segments the data into a canonical dataset and separate edit datasets. This approach provides versioning capabilities with much lower complexity by only storing changes rather than complete dataset copies.
Solution Approach 2:
The patent uses selective copying of only the necessary data components (edits and buffers) rather than copying entire datasets. The query rewriter synthesizes views by joining the canonical dataset with relevant edits, providing versioned access without the overhead of storing multiple complete dataset copies.
3Speed
If multiple data copies are stored for versioning, then query latency is reduced, but the amount of data storage required increases tremendously
Solution Approach 1:
The patent implements versioning by storing only the differences (edits) rather than complete dataset copies. The edits dataset contains only the changes accumulated since the canonical dataset was created, dramatically reducing storage requirements while still enabling versioned queries through the query rewriter.
Solution Approach 2:
The system maintains a canonical dataset as the base version and stores only the necessary edits in the edits dataset. When querying, the system recovers the needed version by applying relevant edits to the canonical dataset, discarding the need to store multiple complete dataset versions.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A database system comprised of a decoupled compute layer and storage layer is implemented to store, build, and maintain a canonical dataset, a temporary buffer, and an edits dataset. The canonical dataset is a set of batch updated data. The data is appended in chunks to the canonical dataset such that the canonical dataset becomes a historical dataset over time. The buffer is a write ahead log that contains the most recent chunks of data and provides atomicity and durability for the database system. The edits dataset is the set of data that contains edits such as cell mutations, row appends and/or row deletions. The database system enables users to make cell or row-level edits to tables and observe those edits in analytical systems or downstream builds with minimal latency.