Elastic Cloud Storage Object Replication Tracking via Journal Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Elastic Cloud Storage (ECS) faces challenges in tracking replication progress at the object level due to its replication mechanism operating at the chunk level, making it difficult to determine when objects have arrived at a second zone and register their arrival without additional checks or interactions, especially in geographically distributed setups.
Innovation Solution
Implementing a method that analyzes metadata and journal entries to track object arrival by checking repository chunks and journal chunks, allowing the blob service to register objects and update the chunk table, thereby facilitating minimal per-object checks and interactions between storage services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If replication operates at the chunk level, then write performance and capacity efficiency are improved, but object-level replication progress tracking becomes difficult
Solution Approach 1:
The patent introduces journal chunks as intermediary structures that record object-level replication metadata. These journal chunks act as a mediator between the chunk-level replication mechanism and object-level tracking requirements, allowing the system to maintain high write performance while enabling visibility into replication progress at the object level through the journal entries that track which chunks have been replicated for each object
Solution Approach 2:
The patent adds a new dimension to the replication tracking system by introducing journal chunks that store metadata about replication progress. This creates an additional layer of information structure that enables object-level tracking without interfering with the existing chunk-level replication operations, effectively adding a tracking dimension alongside the data storage dimension
2Loss of information
If additional checks are performed to track object arrival, then replication progress visibility is improved, but system interactions and overhead increase
Solution Approach 1:
The patent performs preliminary action by recording replication progress information in journal chunks as chunks are replicated. This advance recording of metadata means that when object-level tracking is needed, the information is already available in the journal entries, eliminating the need for additional checks or interactions to determine replication status
3Loss of information
If object arrival is tracked at the object level, then customer visibility requirements are met, but the replication mechanism complexity increases
Solution Approach 1:
The patent creates a copy of the replication tracking function by introducing journal chunks that mirror the chunk-level replication information from the perspective of objects. Instead of modifying the chunk-level replication mechanism, the system creates a parallel journal structure that copies and organizes replication status information suitable for object-level tracking, thereby meeting visibility requirements without increasing the core replication mechanism's complexity
Data Source
AI summary
Elastic Cloud Storage (ECS) uses a specific method for disk capacity management, where disk space is partitioned into a set of blocks of fixed size called chunks. Chunk data can comprise user data and all kinds of metadata. The replication mechanism implemented in ECS works at the chunk level. However, object level data can be used to provide visibility for client applications. Thus, object arrival can be determined to track replication progress at the object level for systems with a replication mechanism that works at the chunk level.