Log Sequencer Assigning Sequence Numbers to Distributed Data Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current log storage services in distributed computing systems face inefficiencies in write availability and tolerance to spikes in writes due to limited storage systems and single-point failures in record placement applications, leading to potential data loss and delays.
Innovation Solution
A log device with a log sequencer that assigns unique sequence numbers to records and determines storage nodes using a non-deterministic method, decoupling sequence number assignment from storage, allowing records to be stored across a larger pool of nodes, enhancing write availability and tolerance to write spikes, and enabling efficient resource deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the log is stored on a small number of storage systems (e.g., three), then the device complexity is reduced, but the reliability deteriorates because any single storage system failure can lead to data loss
Solution Approach 1:
The patent segments the log storage across multiple storage systems, where each storage system stores only a portion of the log records. The record placement application divides the log into segments and distributes them across N storage systems, allowing the system to tolerate failures of individual storage systems while maintaining overall reliability and write availability.
2Adaptability or versatility
If record placement applications are used to determine storage system selection based on mathematical functions, then the adaptability improves, but the reliability deteriorates because the record placement application becomes a single point of failure
Solution Approach 1:
The patent extracts the record placement application from the storage system and implements it as a separate, distributed component. Multiple record placement applications are deployed across different storage systems, eliminating the single point of failure. The placement logic is distributed rather than centralized, so the failure of one record placement application does not affect the overall system availability.
3Device complexity
If the log is stored on a small number of storage systems, then the device complexity is reduced, but the productivity deteriorates because the storage systems cannot tolerate spikes in writes
Solution Approach 1:
The patent segments the log storage across multiple storage systems, where each storage system stores only a portion of the log records. This segmentation allows write operations to be distributed across multiple storage systems simultaneously, increasing the overall write throughput and enabling the system to tolerate spikes in write volume without overloading individual storage systems.
4Reliability
If multiple replicas of the log are stored across different storage systems, then the reliability improves, but the device complexity increases
Solution Approach 1:
The patent implements a universal record placement application that can determine the storage location of any log record using a consistent mathematical function. This placement application serves multiple functions: it distributes records across storage systems, enables efficient retrieval, and maintains data durability. The same placement logic is used for both writing and reading operations, simplifying the overall system architecture while maintaining reliability.
Data Source
AI summary
The disclosure is directed to storing a data log file (“log”) in a distributed computing system. The log can be stored as a sequence of data records (“records”). A number of writer computers (“writers”) can write to a log, and a number of reader computers (“readers”) can read from the log. A log sequencer computer (“log sequencer”) facilitates storage of the records in data storage nodes of the distributed computing system. When the log sequencer receives a request for appending a record to a log, the log sequencer assigns a sequence number to the record that is unique and increases monotonically within the log. When a reader requests the log, the records can be delivered to the reader in the order of their sequence numbers. Further, since the records are associated with sequence numbers, a reader can seek to arbitrary sequence number within the log.