Delta Log Space Reclamation via Bookmarks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The configuration of delta log infrastructure in storage clusters with multiple delta log instances and a single raw delta log faces challenges such as increased write amplification and recovery complexity, as well as difficulties in safely reclaiming allocated space in the single raw delta log.
Innovation Solution
The solution involves writing delta updates for different metadata types to respective metadata delta logs optimized for each type, switching active and de-staging designations of data containers, and using bookmarks to manage memory space reclamation, ensuring that space is reclaimed only when bookmarks are determined to be the oldest in the list, thereby deferring reclamation until necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple raw delta logs are provided for different metadata types, then write amplification increases and recovery complexity increases, but if a single raw delta log is used, then space reclamation becomes unsafe
Solution Approach 1:
The patent introduces bookmarks as intermediary markers within the single raw delta log. Each bookmark records the position and metadata type, allowing the system to track which metadata types have been de-staged. This intermediary mechanism enables safe space reclamation by preventing deletion of log entries that might still be needed for recovery, while avoiding the complexity of multiple separate logs.
Solution Approach 2:
The patent changes the parameter of the raw delta log from multiple separate logs to a single unified log. This parameter change is accompanied by introducing a new parameter (bookmarks) that tracks the state of different metadata types within the single log, thereby achieving both simplified structure and safe reclamation.
2Device complexity
If a single raw delta log is used for multiple metadata types, then device complexity is reduced, but write amplification increases
Solution Approach 1:
The patent segments the single raw delta log into different regions or sections corresponding to different metadata types, marked by bookmarks. This segmentation allows the system to treat different metadata types separately within a unified structure, enabling selective de-staging and reducing unnecessary write amplification by only processing de-staging for specific metadata types when their data containers are full.
3Productivity
If multiple raw delta logs are used for different metadata types, then space reclamation can be performed independently for each type, but recovery mechanisms become highly complex
Solution Approach 1:
The bookmark mechanism serves as an intermediary that enables independent tracking and reclamation of space for different metadata types within a single log. Each bookmark acts as a boundary marker that allows the system to identify and reclaim space for specific metadata types without affecting others, thereby maintaining reclamation efficiency while avoiding the complexity of multiple separate logs.
Data Source
AI summary
Techniques for handling journal space in a storage cluster with multiple delta log instances. The techniques include writing delta updates for a respective metadata type to an “active” set of data containers in a delta log instance and raw delta updates to a raw delta log, switching a designation of the “active” set of data containers from “active” to “de-staging” once one or more of the “active” set of data containers has been filled, writing a bookmark for the respective metadata type to the raw delta log and a bookmark list, determining that a de-staging operation has been completed for writing the delta updates from the “de-staging” set of data containers to a storage array, determining that the bookmark for the respective metadata type is the oldest bookmark in the list, and reclaiming space between a tail of the raw delta log and the bookmark written to the raw delta log.