Automatic Data Store Log Archiving via Fleet Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud-based data storage systems lack efficient mechanisms for long-term archiving of data store log data, leading to challenges in accessing historical changes to data objects, which is crucial for offline data mining and trend analysis.
Innovation Solution
The implementation of a system that automatically archives data store log data by replicating log entries to a separate archive using a fleet of workers, allowing clients to access archived records directly with low-cost storage resources, and providing long-term storage without expiration dates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log data is stored in the primary data store for long-term access, then historical change data is accessible, but storage costs increase and data access impacts primary store performance
Solution Approach 1:
The patent extracts historical change data from the primary data store by implementing a separate archiving mechanism. Log entries are copied to a dedicated archive storage system, separating the function of primary data storage (for active transactions) from historical data retention (for analysis and compliance). This extraction resolves the contradiction by maintaining access to historical changes while reducing the storage burden on the primary data store.
Solution Approach 2:
The patent creates a copy of log data in a separate archive storage system. Instead of storing all historical data in the primary store, the system copies relevant log entries to an archive, allowing long-term retention without consuming primary storage resources. This copying mechanism enables historical data accessibility while optimizing storage resource allocation.
2Productivity
If a fleet of workers is used to replicate log data to archive, then archiving speed increases, but system complexity increases
Solution Approach 1:
The patent segments the archiving function into multiple independent worker processes that can operate in parallel. Each worker handles specific replication tasks, allowing the system to scale archiving capacity by adding more workers without redesigning the core architecture. This segmentation achieves high archiving speed while managing complexity through modular design.
Solution Approach 2:
The archiving system operates autonomously using a fleet of workers that automatically replicate log data without requiring manual intervention. The system self-manages the archiving process, selecting which log entries to archive and distributing replication tasks across workers, thereby achieving high productivity with controlled complexity through automation.
3Quantity of substance
If log data is archived to long-term storage, then storage cost decreases, but access time to archived data increases
Solution Approach 1:
The patent applies different storage characteristics to different data types: hot data (recent changes) remains in the primary data store for fast access, while cold data (historical changes) is archived to cost-effective long-term storage. This local quality differentiation optimizes the trade-off between access speed and storage cost by placing data where it is most needed.
Solution Approach 2:
The system performs preliminary archiving of log data before it is no longer immediately needed. By proactively copying historical data to the archive, the system prepares cost-effective storage locations in advance, allowing rapid retrieval when historical data is needed while maintaining the cost benefits of long-term storage.
Data Source
AI summary
Methods, systems, and computer-readable media for automatic archiving of data store log data are disclosed. One or more operation records in a log are selected for archival. The one or more operation records comprise data indicative of operations performed on one or more data objects of a data store. The one or more operation records are selected for archival prior to deletion from the log. The one or more operation records are replicated from the log to an archive. Based at least in part on the replicating, the one or more operation records in the log are marked as archived. Based at least in part on the marking as archived, the deletion of the one or more operation records from the log is permitted.