Distributed Object Storage with Bidirectional Chunk References
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional object storage systems face scalability limitations due to the reliance on a single metadata server for managing chunk storage resources, which hinders efficient distribution of metadata across multiple servers and complicates allocation of storage resources.
Innovation Solution
A distributed object storage system that uses a manifest tracking subsystem to encode forward-references from version manifests to chunks and a chunk payload subsystem to encode back-references, allowing for efficient cross-referencing without specifying storage locations, enabling a many-to-many relationship between chunks and version manifests and reducing the need for tight metadata server synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single metadata server is used to manage chunk storage resources, then the system structure is simple, but the scalability is limited and synchronization complexity increases
Solution Approach 1:
The patent divides the metadata management function into two separate subsystems: the manifest tracking subsystem that manages version manifests and the chunk payload subsystem that manages chunk storage. This segmentation allows each subsystem to be independently scaled and managed, resolving the contradiction between simple structure and scalability by distributing metadata responsibility across multiple servers while maintaining clear functional boundaries.
2Productivity
If metadata is distributed across multiple servers, then scalability is improved, but synchronization complexity increases
Solution Approach 1:
The patent extracts the cross-reference information from the metadata management responsibility and places it directly in the chunk objects themselves. The chunk payload subsystem encodes back-references to version manifests within the chunk data, eliminating the need for the manifest tracking subsystem to actively track and synchronize chunk locations. This extraction resolves the synchronization complexity by making the system self-describing.
Solution Approach 2:
The chunk objects contain embedded back-references that allow them to self-identify their associated version manifests without external metadata server intervention. This self-service mechanism enables the distributed subsystems to operate independently while maintaining consistency, as the chunk data itself carries the necessary reference information for reconstruction and validation.
3Productivity
If forward-references and back-references are encoded between manifests and chunks, then storage efficiency is improved, but the data structure complexity increases
Solution Approach 1:
The patent merges the forward-reference from the manifest to the chunk with the back-reference from the chunk to the manifest into a unified cross-reference system. By encoding both reference directions, the system enables efficient storage operations where chunks can be independently stored and retrieved without requiring complete manifest information, while the manifest can reference chunks without needing to know their physical locations. This merging resolves the structure complexity by creating a bidirectional reference system that simplifies overall data management.
Data Source
AI summary
The present disclosure provides an advantageous systems and related methods for scalable object storage. One embodiment relates to a system that includes at least a manifest tracking subsystem and a chunk payload subsystem. The manifest tracking subsystem stores version manifests and encodes forward-references from the version manifests to chunks. The chunk payload subsystem stores the chunks and encodes back-references to version manifests that reference the chunk. Other embodiments, aspects and features are also disclosed.