Stream Session Backreferences for Distributed Object Storage Chunk Retention

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional object storage systems face scalability constraints due to reliance on centralized metadata servers for tracking chunk allocation, which limits their ability to efficiently manage and reclaim storage resources, especially in systems with frequent data turnover like virtual disks and databases.

Innovation Solution

The implementation of stream sessions and stream session backreferences allows for distributed tracking of chunk retention, enabling efficient reclamation of resources by associating backreferences with specific retention epochs and reducing the need for precise tracking of chunk versions, thus improving scalability and resource management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If centralized metadata servers are used to track chunk allocation, then chunk tracking precision is improved, but system scalability deteriorates

Engineering Contradiction:

Improvechunk tracking precisionVSAvoidsystem scalability

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The patent divides the centralized metadata tracking function into distributed stream session backreferences across multiple servers. Each server maintains backreferences for specific stream sessions rather than a single centralized server tracking all chunks, enabling parallel processing and system scaling while maintaining retention tracking capability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts the chunk tracking function from the metadata server by implementing stream session backreferences that are maintained independently by storage servers. This separates the retention tracking responsibility from centralized metadata management, allowing the metadata server to scale independently from chunk tracking operations

Inventive Principle:

Principle #2Taking out (Extraction)

2Measurement precision

If precise tracking of chunk versions is maintained, then retention accuracy is improved, but computational complexity deteriorates

Engineering Contradiction:

Improveretention accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements dynamic stream sessions with finite retention epochs that automatically expire. Instead of maintaining permanent precise tracking of all chunk versions, the system uses time-limited stream sessions where backreferences are created during active editing and automatically removed when the retention epoch expires, reducing computational complexity while maintaining accuracy during active retention periods

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent uses periodic stream session epochs to refresh and maintain retention tracking. Rather than continuous tracking, the system periodically creates new stream sessions for active editing operations, allowing old sessions to expire and be garbage collected, thus reducing the accumulated computational burden of tracking

Inventive Principle:

Principle #19Periodic action

3Adaptability or versatility

If stream session backreferences are used instead of precise chunk version tracking, then system scalability is improved, but retention precision deteriorates

Engineering Contradiction:

Improvesystem scalabilityVSAvoidretention precision

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent merges multiple version manifests from the same stream session into a single backreference entry. Instead of creating individual backreferences for each version manifest, the system consolidates them by stream session, reducing the number of backreferences while maintaining retention accuracy for all versions within that session through the shared stream session identifier and epoch

Inventive Principle:

Principle #5Merging (Combining)

4Adaptability or versatility

If distributed metadata servers are used, then system scalability is improved, but coordination overhead deteriorates

Engineering Contradiction:

Improvesystem scalabilityVSAvoidcoordination overhead

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements self-service distributed metadata management where each server independently maintains stream session backreferences for its local stream sessions without requiring coordination with other servers. The stream session identifier and epoch provide self-contained retention information that can be validated and processed independently, eliminating the need for inter-server coordination protocols

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10834165B2Chunk retention in a distributed object storage system using stream sessions and stream session backreferences

Publication Date: 2020.11.10 NEXENTA BY DDN INC

AI summary

The present disclosure provides an advantageous systems and related methods for scalable object storage. In accordance with an embodiment of the invention, a stream session is opened to initiate a retention epoch during which a series of edits to an object by a source may be performed without seeing edits from other stream sessions. Backreferences identifying the stream session (i.e. stream session backreferences) are applied to retained chunks. While a payload chunk has one or more such associated stream session backreferences, the chunk is retained and not expunged. Advantageously, transitory chunks that are put during the retention epoch may be unput promptly so as to release valuable cache resources. In comparison, the prior approach only eventually releases the data storage resources for expunged version manifests. Other embodiments, aspects and features are also disclosed.