Append-Only Data Store Version Resolution via Local Side Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems face challenges in efficiently handling updates for key values, especially when dealing with prior versions and non-recent data, due to limitations in random access and in-place updates in read-friendly storage systems.
Innovation Solution
A method is implemented where transactional operations are processed by maintaining transactional operations in local side logs, with parallelism across nodes for handling updates, and a FirstKnownStartTimeIndex is used to manage updates and retire rows, allowing for efficient processing and patching of prior versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If updates are performed in conventional data management systems using read-friendly storage systems, then data availability is improved, but random access efficiency deteriorates
Solution Approach 1:
The system segments data into multiple versions stored in an append-only data store, with each version representing a point in time. This segmentation allows efficient random access to any version while maintaining data availability through multiple replicas, resolving the contradiction between reliability and access speed.
Solution Approach 2:
The system performs preliminary actions by pre-computing and storing data in an immutable append-only format with versioning. This preliminary structuring enables efficient random access to historical versions without requiring mutable storage operations, thus improving both availability and access efficiency.
2Ease of operation
If conventional storage systems are used for data management, then ease of operation is improved, but processing speed for updates deteriorates
Solution Approach 1:
The system introduces an intermediary append-only data store that sits between the conventional storage system and the update operations. This intermediary handles all update operations through appends only, maintaining ease of operation while dramatically improving processing speed by eliminating random write operations on conventional storage.
Solution Approach 2:
The system replaces the mechanical random read-write operations on conventional storage with an append-only operational model. This substitution eliminates the performance bottleneck of random writes while maintaining operational simplicity through standardized append operations, thus improving productivity without sacrificing ease of operation.
3Productivity
If parallelism is implemented across nodes, then processing speed is improved, but device complexity increases
Solution Approach 1:
The system segments data across multiple nodes in a distributed architecture, with each node handling a portion of the data. This segmentation enables parallel processing of update operations across nodes, improving processing speed while keeping individual node complexity manageable through consistent partitioning strategies.
Data Source
AI summary
One embodiment provides for a method including performing, by a processing thread, a process that analyzes transactional operations by maintaining the transactional operations in transaction local side logs, and waiting until a successful transaction commit to append the transaction local side logs to a log stream. The processing thread processes the transactional operations on a key used to determine whether existing data is found for the key. The transactional operations are sped up through parallelism based on partitioning tables across nodes handling the transactional operations. A first process is performed by a first processor that processes updates for values of a key based on updating a first start time table index using unique keys and a start time field of a row for a first appearance of each unique key from the transactional operations.