Bypassing Snapshot Data Access in Object Store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication and retrieval methods in multi-site environments, such as archival storage systems, suffer from increased latency and decreased throughput due to the need for data to flow through an intermediary site, which also performs unnecessary transformations like compression/decompression, creating bottlenecks.
Innovation Solution
A bypassing technique that allows direct access to an object store by providing object identifiers and access information to the client, enabling the client to pack and transfer snapshot extents directly, eliminating the intermediary service from the data path and reducing the need for multiple data hops.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data flows through an intermediary site for access to archived data, then the system provides centralized management and indexing capabilities, but latency increases and throughput decreases
Solution Approach 1:
The patent introduces a caching layer at the edge location that stores recently accessed snapshot data and metadata. This caching intermediary reduces latency by serving data locally without requiring trips to the remote archive, while the remote site maintains centralized management. The cache acts as a local intermediary that bridges the gap between centralized control and fast local access.
Solution Approach 2:
The system segments data access operations into two parts: metadata management remains centralized at the remote site, while actual data retrieval is handled locally through caching. This segmentation allows the system to maintain centralized control for indexing and coordination while enabling fast local access for data retrieval, thus reducing latency without sacrificing management capabilities.
2Reliability
If data flows through an intermediary site, then centralized control is maintained, but throughput decreases due to the bottleneck
Solution Approach 1:
The caching layer at the edge location serves as an intermediary that handles data retrieval operations locally, bypassing the bottleneck at the remote site. The cache stores frequently accessed data and can serve multiple requests without requiring communication with the remote archive, thus maintaining centralized control while dramatically improving throughput.
Solution Approach 2:
The system performs preliminary actions by pre-caching snapshot data and metadata at the edge location before they are needed. This preliminary caching of data and indexing information enables fast local retrieval without requiring real-time communication with the remote site, thereby maintaining throughput while preserving centralized management.
3Quantity of substance
If an intermediary site performs data transformations like compression/decompression, then data is optimized for storage, but unnecessary processing increases latency
Solution Approach 1:
The patent extracts the data transformation operations (compression/decompression) from the data retrieval path. Data is pre-compressed when archived at the remote site, and the cache stores the compressed data directly. Retrieval operations only require decompression if the data is being returned to the user, not during every access operation, thus eliminating unnecessary processing latency while maintaining storage efficiency.
Solution Approach 2:
Data transformations such as compression are performed in advance when data is archived at the remote site, before it is cached at the edge location. This preliminary transformation eliminates the need for repeated compression/decompression cycles during data retrieval, reducing processing latency while preserving the storage efficiency benefits of compression.
4Reliability
If multiple data hops are required to access snapshot data, then centralized management is achieved, but the number of transmission steps increases latency
Solution Approach 1:
The caching layer at the edge location acts as an intermediary that stores snapshot data locally, eliminating the need for multiple data hops between the user and the remote archive. The cache provides a local endpoint for data access, reducing the number of transmission steps from multiple hops to a single local retrieval operation, thus decreasing latency while maintaining management coordination.
Solution Approach 2:
The system segments the data access architecture into a remote management layer and a local data retrieval layer. The remote site handles metadata management and coordination, while the local cache handles actual data retrieval. This segmentation reduces the number of data transmission hops by serving data from the local layer, thereby decreasing access time while preserving centralized management capabilities.
Data Source
AI summary
A bypassing technique bypasses an indexing service and provides a bypass data path for transferring/retrieving snapshots from a production cluster to an object store. In an embodiment, the production cluster may determine how extents of the snapshots are packed into objects of the object store and transfers the snapshots directly to the object store over the bypass data path. Once the snapshot transfer is completed, the production cluster provides location metadata as to how the snapshot extents are packed into objects to the indexing service. The indexing service is invoked to create an index of the location metadata and is not involved in the data transfer of the snapshots. In another embodiment, the production cluster identifies a snapshot to restore and queries the indexing service to compute the deltas between the snapshot to be restored and a reference snapshot. The indexing service returns a set of segments that indicates the changed delta regions between the two snapshots. The production cluster then issues a direct read to the object store over the bypass data path to read the changed delta segments.