Snapshot Isolation for Big Data Table Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Updating big data tables is complex due to the large volume of data and concurrent processes accessing it, leading to inconsistencies and the need for resource-intensive audit logs to maintain data consistency.
Innovation Solution
The use of snapshot isolation methods, where a coordinator locks metadata, updates data objects, and only commits changes if all updates are successful, ensuring that read processes access a consistent version of the data by rolling back to the last known version in case of failures, and maintaining locks to prevent other processes from updating until the operation is complete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods are used to update big data tables with concurrent processes, then data consistency can be maintained through audit logs, but storage requirements and computational resources increase significantly
Solution Approach 1:
The patent extracts the version information from the actual data objects and stores it separately in a version map data structure. This separation allows the system to maintain data consistency information without storing complete audit logs for every data object, significantly reducing storage requirements while preserving the ability to detect and resolve consistency issues
Solution Approach 2:
The patent segments the data storage system into distinct components: the main data objects stored in HDFS and a separate version map stored in memory. This segmentation allows independent management of data and version information, enabling efficient consistency checking without requiring centralized audit log storage for all data modifications
2Reliability
If locks are maintained on metadata during update operations, then data consistency is ensured, but other processes are blocked until the operation completes
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing version numbers for each data object in the version map before actual update operations begin. This allows the system to quickly verify consistency during updates without requiring lengthy metadata locking, as the version information is already prepared and readily accessible
Solution Approach 2:
The version map acts as an intermediary data structure that mediates between the data objects and the update processes. Instead of locking metadata directly, the system uses version numbers from the version map to track and verify consistency, allowing concurrent processes to proceed without blocking while maintaining data integrity through version comparison
3Reliability
If the system checks version numbers for each data object during updates, then consistency can be verified, but the update process becomes more complex
Solution Approach 1:
The system implements self-service by having data objects automatically include version numbers as intrinsic properties, and the version map automatically track these versions. This self-managing approach eliminates the need for complex external verification mechanisms, as the version information is inherently embedded in the data structure and can be verified through simple comparison operations
Data Source
AI summary
Methods, systems, and computer program products are described herein for updating big data tables using snapshot isolation. The tables may store data (e.g., as data objects) collected from different users. The tables are stored in a first data store. The data's metadata is stored in a second data store. The metadata comprises version numbers for the objects that correspond to a consistent version thereof. When an update process for the table is initiated, a coordinator locks the metadata, and. the objects are updated. If the update to the objects is successful, the coordinator commits the updates and obtains the new version numbers of the objects. If all the commits are successful, the coordinator atomically updates the metadata with the new version numbers and releases the lock. However, if a commit to an object is unsuccessful, a subsequent process rolls back the committed objects to the version numbers specified by the metadata.