Distributed Operation Log for Storage Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems experience significant latency due to the requirement for a 'commit' confirmation before allowing clients to proceed with processing, especially in distributed or virtualized systems with varying storage performance levels, and the need for data replication exacerbates this issue.
Innovation Solution
Implementing a storage system where data writes are initially performed to a fast operation log, allowing immediate confirmation to the client, and then flushed to slower back-end storage devices over time, with a distributed operation log for replication to multiple locations, reducing latency and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data writes are performed to back-end storage devices before confirmation, then data persistence is ensured, but client latency increases significantly
Solution Approach 1:
The patent segments the storage system into a fast operation log (journal) and slower back-end storage devices. Write operations are first committed to the operation log, which provides immediate confirmation to clients, while asynchronous background processes handle the flushing of data to back-end storage devices. This segmentation allows the system to provide both immediate confirmation and eventual data persistence.
Solution Approach 2:
The operation log acts as an intermediary between the client and the back-end storage devices. It temporarily holds write operations and provides confirmation to clients before the data is actually written to the slower back-end storage. This intermediary layer decouples the client's write confirmation from the actual persistence operation, reducing latency while maintaining reliability.
2Reliability
If data is replicated to multiple storage locations for fault tolerance, then system reliability improves, but write latency increases due to synchronization requirements
Solution Approach 1:
The patent implements preliminary action by first writing data to the operation log and providing confirmation to the client before attempting replication to other storage locations. The replication process occurs asynchronously in the background, allowing the system to provide immediate confirmation while eventually achieving the desired fault tolerance through replication.
Solution Approach 2:
The operation log serves as a copy mechanism that immediately replicates write operations to multiple locations in the storage system. This copying approach allows the system to provide fault tolerance through replication while maintaining performance, as the copying occurs in the background without blocking the client's write operation.
3Reliability
If synchronous replication is performed across distributed storage devices, then data consistency is maintained, but system performance deteriorates
Solution Approach 1:
The patent implements dynamic replication where the system adapts its replication strategy based on operational needs. Critical write operations are immediately confirmed based on operation log commitment, while replication to distributed storage devices occurs asynchronously. This dynamic approach allows the system to maintain data consistency while avoiding the performance penalty of synchronous replication.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining the operation log as a continuous write target that provides immediate confirmation, while replication to distributed storage devices continues asynchronously in the background. This continuous operation allows the system to maintain both data consistency and high performance simultaneously.
Data Source
AI summary
Disclosed is an improved approach for implementing a storage system having an operation log where data writes are initially performed to a fast operation log, rather than to its permanent storage location. A distributed operation log can be employed to replicate the data to multiple locations in the storage system.