Blob Store Transaction Log for Concurrent Dataset File Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data storage systems face inefficiencies in metadata management for large data stores, leading to slowed transaction speeds due to unwieldy metadata and inconsistent data states during concurrent modifications.
Innovation Solution
A big data metadata protocol that maintains a transaction log with ordered logs of atomic actions, using optimistic concurrency control to ensure transactional guarantees and detect conflicts, allowing parallel processing and efficient querying and updating of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional metadata management is used for large data stores, then data storage capacity is maintained, but transaction speed deteriorates due to unwieldy metadata
Solution Approach 1:
The patent segments the monolithic metadata management system into distributed metadata servers that handle different portions of metadata independently. This segmentation allows parallel processing of metadata operations, improving transaction speed while managing the complexity of large data stores through distributed architecture.
Solution Approach 2:
The patent introduces a hierarchical dimension to metadata management by organizing metadata into multiple levels (e.g., namespace-level, dataset-level, file-level). This dimensional organization reduces the complexity any single metadata server must handle while maintaining comprehensive tracking of all data operations.
2Productivity
If concurrent modifications are allowed for improved productivity, then data consistency deteriorates due to conflicting transactions
Solution Approach 1:
The patent implements preliminary conflict detection by having transactions declare their intended metadata modifications before execution. The system checks for conflicts with ongoing transactions in advance, allowing safe concurrent execution while preventing inconsistent states. This preliminary action enables high productivity without sacrificing reliability.
Solution Approach 2:
The patent employs feedback mechanisms where the metadata server continuously monitors transaction states and provides real-time information about conflicts to clients. This feedback loop allows the system to maintain data consistency by informing transactions of conflicts and coordinating their execution, thereby preserving reliability while allowing concurrent operations.
3Reliability
If metadata tracking is comprehensive for all data operations, then data reliability is improved, but transaction processing time increases
Solution Approach 1:
The patent applies local quality by having different metadata servers specialize in tracking different types of operations or different portions of the data namespace. This specialization allows each server to optimize its tracking mechanisms for its specific domain, maintaining comprehensive reliability while reducing the time penalty through localized, efficient processing.
Data Source
AI summary
A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.