Local Delta Counters for Distributed Storage Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronizing global counters across multiple nodes in a distributed data storage system is challenging due to expensive communication overhead, especially when updates occur frequently.
Innovation Solution
Each processing node maintains local in-memory delta counters to track changes to global counters, with a summarization process merging delta count values into global counters periodically or on demand, minimizing cross-node communication and ensuring consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If global counters are synchronized frequently across multiple nodes, then counter consistency is improved, but communication overhead increases
Solution Approach 1:
The patent divides the global counter into two parts: a persistent global counter stored in non-volatile memory and in-memory delta counters that track changes. This segmentation allows nodes to update delta counters locally without immediate communication, reducing communication overhead while maintaining consistency through periodic synchronization.
Solution Approach 2:
The patent implements preliminary action by maintaining in-memory delta counters that pre-track all changes before synchronization. Nodes accumulate delta counter values locally and only communicate the net changes to the global counter, rather than synchronizing after every individual update, thus reducing communication frequency and overhead.
2Productivity
If communication between nodes is minimized, then system performance is improved, but counter synchronization difficulty increases
Solution Approach 1:
The patent introduces an intermediary mechanism in the form of in-memory delta counters that mediate between local node operations and global counter state. These delta counters serve as a buffer that simplifies synchronization by capturing all changes locally and enabling batch updates to the global counter, reducing the complexity of real-time coordination.
Solution Approach 2:
The patent implements periodic synchronization where in-memory delta counters are flushed to the global counter at scheduled intervals or upon specific triggers. This periodic action reduces communication frequency compared to continuous synchronization, improving performance while managing complexity through predictable, periodic updates rather than complex real-time coordination.
3Loss of energy
If in-memory delta counters are used to track changes, then communication overhead is reduced, but memory usage increases
Solution Approach 1:
The patent applies local quality by implementing in-memory delta counters only at each node to track local changes, rather than maintaining full global counter state at every node. This localized approach reduces overall memory usage compared to replicated state while still enabling reduced communication overhead through local tracking of changes.
Data Source
AI summary
A data storage system performs updating of a set of global counters stored in persistent storage accessible to a plurality of processing nodes of a data storage system, the global counters storing respective global count values describing associated units of data storage shared by the processing nodes for performing data storage operations. The updating includes, by each processing node, using a respective set of in-memory delta counters storing respective delta count values to track changes to respective global count values due to data storage operations performed by the processing node with respect to the units of data storage. A summarization process is regularly executed to merge the delta count values of the delta counters of all processing nodes into the respective global counters.