Hierarchical Coherency Log for Distributed Storage Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed storage management techniques face challenges in reducing management overhead, particularly in asynchronous mirroring schemes, where tracking changes across multiple datasets imposes computational burdens and delays data requests.
Innovation Solution
Implementing a hierarchical change log with multiple sub-logs of varying priority levels, where higher priority sub-logs are optimized for faster searching or writing, and lower priority sub-logs for greater detail, to reduce the overhead of maintaining coherence across mirrored datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous mirroring is used to maintain data coherency, then data consistency across mirrors is improved, but storage performance and throughput deteriorate due to increased overhead and connection burden
Solution Approach 1:
The patent segments the change tracking mechanism into multiple sub-logs organized in a hierarchical structure. Each sub-log tracks changes for specific data ranges or priorities, allowing parallel processing and reducing the bottleneck of centralized change tracking. This segmentation enables asynchronous mirroring to maintain coherency without synchronous performance penalties.
2Productivity
If asynchronous mirroring is used to improve storage performance, then throughput is improved, but data coherency management becomes more complex and requires careful monitoring
Solution Approach 1:
The patent implements preliminary action by pre-establishing a hierarchical change log structure that automatically tracks data changes before mirroring operations occur. The system proactively monitors and records changes in organized sub-logs, so that when asynchronous mirroring is needed, the coherency information is already prepared and structured, eliminating the need for complex real-time monitoring during the mirroring process.
3Reliability
If change tracking is implemented to maintain coherency across mirrors, then data consistency is improved, but processing overhead and bandwidth consumption increase
Solution Approach 1:
The patent applies local quality by organizing change tracking into specialized sub-logs with different characteristics optimized for specific purposes. Some sub-logs may track frequent small changes efficiently, while others track less frequent but more significant changes. This local optimization reduces the overall processing overhead compared to a uniform change tracking approach, as each sub-log can use the most efficient tracking method for its specific data type or priority level.
Data Source
AI summary
A system and method for managing distributed coherent datasets using a hierarchical change log is provided. In some embodiments, a distributed storage system is provided that includes a primary storage device containing a primary dataset and a mirror storage device containing a mirror dataset. The mirror dataset includes a coherent copy of the primary dataset. The distributed storage system further includes a hierarchical change log tracking a coherence state for the mirror dataset. The hierarchical change log includes a first sub-log and a second sub-log, and a block range of the first sub-log overlaps a block range of the second sub-log. The hierarchical change log may define a priority relationship between the first sub-log and the second sub-log governing the overlap. The first sub-log and the second sub-log may be independently configured and may be different in one of a representation and a block size.