Reference Count Propagation in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in tracking reference counts for billions of data objects across a peer-to-peer network, leading to high network traffic and CPU costs, and challenges in determining when objects can be safely deleted due to simplistic algorithms.
Innovation Solution
A method and system for tracking object references across network nodes by maintaining a local reference count (LRC) independent of object instances, allowing negative values, and calculating a global reference count (GRC) to determine safe de-allocation, while enabling flexible placement of objects based on network performance or reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If simplistic algorithms are used to track reference counts for billions of objects, then the implementation is easier, but network traffic and CPU costs become unacceptable
Solution Approach 1:
The patent segments the reference count tracking into local reference counts maintained at each node independently, rather than requiring global tracking. Each node maintains its own LRC for objects it knows about, eliminating the need for complex global coordination and reducing network traffic significantly.
Solution Approach 2:
The patent uses copies of reference count information stored locally at each node. Instead of maintaining a single global reference count, each node keeps a local copy (LRC) of the reference count for objects in its scope, allowing independent tracking without global synchronization overhead.
2Reliability
If reference counts are tracked for all objects, then object de-allocation can be determined, but the complexity of tracking increases significantly
Solution Approach 1:
The patent applies local quality by making each node maintain reference counts only for objects within its local scope (LRC), rather than tracking all objects globally. This localized approach reduces the complexity at each node while maintaining reliable de-allocation determination through the global reference count mechanism.
Solution Approach 2:
The patent merges local reference counts from multiple nodes to determine the global reference count. By combining LRCs from all nodes, the system achieves reliable object de-allocation determination without requiring any single node to track all objects individually, thus reducing overall system complexity.
3Ease of operation
If objects are placed on nodes based on object name, then placement is deterministic, but flexibility for network performance optimization is reduced
Solution Approach 1:
The patent introduces dynamics by allowing object placement to be flexible rather than fixed. Objects can be placed on any node based on current network conditions, performance considerations, or load balancing requirements, while the reference count tracking mechanism adapts to handle this flexibility through local reference count maintenance.
Data Source
AI summary
Methods and systems are provided for tracking object instances stored on a plurality of network nodes, which tracking enables a global determination of when an object has no references across the networked nodes and can be safely de-allocated. According to one aspect of the invention, each node has a local object store for tracking and optionally storing objects on the node, and the local object stores collectively share the locally stored instances of the objects across the network. One or more applications, e.g., a file system and/or a storage system, use the local object stores for storing all persistent data of the application as objects.