Streaming Transaction Mechanism for Low-Latency Distributed Database Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transaction-based systems in distributed database systems are inefficient for low-latency data requirements, particularly in financial market pricing datasets that need real-time updates, as they require frequent opening and committing of transactions, which is computationally expensive and inefficient.
Innovation Solution
Implementing a streaming transaction mechanism that allows updates to be applied to datasets without incrementing the version number during the transaction, enabling real-time data access and aggregation of updates until the transaction is committed, thereby reducing computational overhead and improving latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If frequent transactions are opened and committed to provide real-time data updates, then data freshness is improved, but computational cost increases
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing transaction snapshots at regular intervals. When real-time data is needed, the system can retrieve these pre-computed snapshots without needing to execute and commit a new transaction, thus providing fast data access while avoiding the computational overhead of frequent transaction commits.
Solution Approach 2:
The system dynamically adjusts the transaction commit frequency based on data freshness requirements. For less time-critical data, snapshots are taken at standard transaction intervals. For time-critical data, the system can interpolate between snapshots or use change data capture to provide near-real-time views without committing transactions at maximum frequency, optimizing the balance between speed and computational cost.
2Loss of time
If transaction commits are performed at the speed of data change to deliver real-time data, then data latency is reduced, but system efficiency deteriorates
Solution Approach 1:
The system segments data access into two paths: (1) For non-time-critical queries, use standard transaction commits at optimized intervals; (2) For time-critical queries, use snapshot retrieval and interpolation techniques. This segmentation allows the system to minimize latency only where necessary while maintaining high overall efficiency for the majority of queries.
Solution Approach 2:
The system creates copies of transaction snapshots at regular intervals and stores them in a dedicated snapshot storage structure. When real-time data is needed, the system retrieves these pre-made copies rather than executing new transactions, significantly reducing latency for time-critical operations while avoiding the efficiency penalty of frequent commits.
3Reliability
If a transaction-based system is used to ensure data consistency, then data reliability is improved, but responsiveness to streaming updates deteriorates
Solution Approach 1:
The system performs preliminary actions by capturing and storing transaction snapshots at regular intervals while maintaining the integrity of the transaction-based system. These pre-captured snapshots provide a reliable baseline that can be quickly retrieved without compromising data consistency, enabling fast access while preserving reliability guarantees.
Solution Approach 2:
The system introduces snapshots as an intermediary layer between the transaction-based storage system and the data access layer. Snapshots provide a consistent view of data at specific points in time without requiring active transactions to be open, thus mediating between the reliability requirements of the transaction system and the speed requirements of real-time data access.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for implementing a streaming transaction in a transaction based data storage system is disclosed. In an embodiment, a server computer system stores a dataset comprising a plurality of files where the dataset comprises a corresponding version number. The server computer system receives a command to modify the dataset and, in response, starts a streaming transaction for the dataset. During the streaming transaction, the server computer system receives a plurality of updates to the dataset, wherein each of the plurality of updates modifies one or more files of the plurality of files. The server computer system executes the plurality of updates in the dataset without modifying the corresponding version number of the dataset. When the server computer system commits the streaming transaction to the dataset, the server computer system increments the corresponding version number of the dataset.