Asynchronous Log Storage via Hot and Cold Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed log-structured storage systems face challenges in maintaining consistency across multiple nodes, leading to increased costs and latency in recording log records, particularly with asynchronous updates where log records may not arrive in sequence, causing storage nodes to burden with ordering and acknowledgment delays.
Innovation Solution
Implementing a hot log and cold log storage approach where log records are initially stored in a hot log portion out of sequence and later moved to a cold log portion for sequential storage, using Log Sequence Numbers (LSNs) to maintain order, and employing techniques like coalescing and compression to optimize storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous log updates are used in distributed storage systems, then productivity is improved by allowing parallel log record processing, but log records may be stored out of order increasing device complexity
Solution Approach 1:
The patent segments the storage system into multiple storage nodes, each independently processing log records. Each node maintains local ordering using LSNs while the system overall benefits from parallel asynchronous processing across nodes, resolving the contradiction between throughput and ordering complexity.
Solution Approach 2:
The patent introduces LSNs as an intermediary mechanism that mediates between asynchronous log record arrival and sequential processing requirements. LSNs provide a simple comparison-based ordering mechanism that enables parallel processing while maintaining correctness, reducing the complexity burden.
2Reliability
If synchronous approaches are used to ensure consistency across distributed storage nodes, then reliability is improved, but latency increases due to acknowledgment requirements
Solution Approach 1:
The patent applies preliminary action by pre-assigning LSNs to log records before they are fully processed and stored across the distributed system. This allows the system to maintain consistency guarantees while avoiding synchronous acknowledgment delays, as the ordering is predetermined rather than negotiated in real-time.
Solution Approach 2:
Instead of requiring synchronous acknowledgments to prove consistency, the patent inverts the approach by using LSNs to prove ordering correctness without requiring real-time acknowledgment exchanges. This asynchronous verification method maintains reliability while eliminating the time penalty of synchronous protocols.
3Stability of the object's composition
If log records are stored sequentially to maintain order, then consistency is maintained, but productivity decreases due to sequential write operations
Solution Approach 1:
The patent segments the log storage across multiple storage nodes, allowing parallel sequential writes to different nodes. Each node maintains local sequential ordering for its portion of the log, while the overall system achieves higher throughput through parallelization, resolving the contradiction between sequential ordering and storage speed.
Solution Approach 2:
The patent transitions from a single-dimension sequential storage model to a multi-dimensional model where LSNs provide logical ordering while physical storage occurs in parallel across multiple nodes. This dimensional change allows simultaneous maintenance of order and increase in storage throughput.
Data Source
AI summary
A log-structured data store may implement optimized log storage for asynchronous log updates. In some embodiments, log records may be received indicating updates to data stored for a storage client and indicating positions in a log record sequence. The log records themselves may not be guaranteed to be received according to the log record sequence. Received log records may be stored in a hot log portion of a block-based storage device according to an order in which they are received. Log records in the hot log portion may then be identified to be moved to a cold log portion of the block-based storage device in order to complete a next portion of the log record sequence. Log records may be modified, such as compressed, or coalesced, before being stored together in a data block of the cold log portion according to the log record sequence.