Transaction Log Migration via Read Cache Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face inefficiencies in managing transaction logs and read caches, leading to increased processing and storage resource consumption, potential data loss, and slow recovery processes.
Innovation Solution
Implementing a highly accessible read cache created from transaction log entries, duplicating it across separate nodes for redundancy, and generating snapshots to reduce storage needs, allowing for efficient data migration to secondary or distributed storage systems, thereby minimizing data on fast local storage and ensuring quick recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transaction logs are maintained for a highly accessed large scale distributed storage system, then data recovery capability is improved, but processing and storage resource consumption increases
Solution Approach 1:
The patent extracts only the most recent transaction log entry and replicates it to other storage nodes, rather than maintaining complete transaction logs on all nodes. This selective extraction reduces storage and processing resources while maintaining recovery capability through the read cache reconstruction process.
Solution Approach 2:
The system performs preliminary action by creating and maintaining a read cache that contains the current state of metadata before any failure occurs. This pre-prepared cache enables rapid recovery without requiring full transaction log replay, thus reducing resource consumption during normal operation while ensuring reliability.
2Reliability
If the read cache is duplicated to a separate node, then recovery reliability is improved, but storage resource consumption increases
Solution Approach 1:
The patent creates a read cache that is duplicated across storage nodes, allowing the system to reconstruct the current state of metadata without requiring complete transaction log replication. This copying approach provides redundancy for recovery while consuming less storage resources than full transaction log duplication.
3Reliability
If transaction logs are replayed to recreate current or prior state of metadata, then data recovery is achieved, but processing time increases
Solution Approach 1:
The system maintains a read cache that is continuously updated with the current state of metadata during normal operations. This preliminary preparation of the cache state eliminates the need for time-consuming full transaction log replay during recovery, enabling rapid reconstruction of metadata state.
Solution Approach 2:
Instead of replaying entire transaction logs during recovery, the system extracts and utilizes the pre-computed read cache state, which contains only the necessary information to reconstruct the current metadata state, thus dramatically reducing recovery time.
4Ease of operation
If large amount of metadata is maintained for effective searches, then search capability is improved, but processing and storage resource consumption increases
Solution Approach 1:
The patent extracts the essential current state of metadata into a read cache that is maintained separately from the complete transaction logs. This extraction allows effective search operations on the condensed cache data while the full metadata history remains in the transaction logs, reducing processing and storage resource consumption for search operations.
Data Source
AI summary
A system, method and apparatus for storing metadata in a metadata store in a robust and efficient manner including receiving a request from a client to perform a data transaction, updating a key-value pair in a metadata store based on the request, entering the data transaction in a transaction log, replicating the last transaction log entry in at least one other storage node in the metadata store and, in response to a threshold event, migrating a first portion of the transaction log from the a data storage medium to a second data storage medium of each of the plurality of data storage nodes.