Chain Logging Using Key-Value Data Storage for Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed systems face challenges in maintaining high availability and durability of logging systems, particularly when subsystems like logging systems are offline, leading to potential downtime and loss of sales, as seen in online merchant operations.
Innovation Solution
Implementing a chain logging system using key-value data storage that allows for highly available, low-latency, and durable logging by appending events to a chain with a head and nodes, enabling concurrent access and idempotent additions, and utilizing a distributed hash table for scalable and fault-tolerant storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a traditional logging system is used in distributed systems, then the system structure is simple, but the logging system becomes unavailable when offline, causing downtime and loss of sales
Solution Approach 1:
The logging system is segmented into independent components: event producers that generate log entries, a chain data structure that stores events, and a key-value data store that persists the chain. This segmentation allows the logging system to operate independently from other subsystems, maintaining availability even when other parts of the distributed system are offline.
Solution Approach 2:
A key-value data store acts as an intermediary between the logging system and persistent storage. The chain logging system writes to the key-value store, which handles persistence and recovery. This intermediary layer isolates the logging system from potential failures in the underlying storage infrastructure, improving reliability while managing complexity through a standardized interface.
2Reliability
If a logging system stores all log entries persistently, then data durability is improved, but write latency increases and system performance decreases
Solution Approach 1:
The system performs preliminary actions by pre-allocating chain nodes and preparing the data structure for efficient appends. Events are added to the chain head with pre-allocated memory structures, allowing O(1) append operations. The key-value store is pre-configured with appropriate persistence settings, balancing durability requirements with write performance by only persisting when necessary.
Solution Approach 2:
The system changes persistence parameters dynamically based on operational requirements. The key-value store can adjust between volatile and persistent modes for different chains or time periods. This allows the system to optimize for speed during normal operation and switch to durability mode when needed, reducing average write latency while maintaining data durability.
3Productivity
If the logging system allows concurrent access from multiple event producers, then system scalability is improved, but data consistency and duplicate prevention become more difficult
Solution Approach 1:
The system implements feedback mechanisms where each event is assigned a unique identifier and the chain maintains a record of processed events. Event producers receive feedback about whether their events were successfully logged or if duplicates were detected. This feedback loop enables concurrent producers to maintain data consistency without requiring complex locking mechanisms, preserving high throughput while preventing duplicates.
Solution Approach 2:
The system uses copying techniques where event data is replicated across multiple nodes in the distributed system. Each node maintains a copy of the chain or relevant portions of it. This copying approach allows concurrent access and processing while maintaining consistency through replication, enabling scalability without sacrificing data integrity or duplicate prevention.
4Reliability
If the logging system is designed for high availability, then downtime is reduced, but system complexity and resource requirements increase
Solution Approach 1:
The chain data structure serves multiple functions simultaneously: it stores event data, maintains event ordering, enables duplicate detection, and supports concurrent access. The key-value store provides both volatile and persistent storage capabilities through a single interface. This multi-functionality reduces the number of separate components needed, achieving high availability without proportionally increasing system complexity.
Data Source
AI summary
Methods, systems, and computer-readable media for chain logging using key-value data storage are disclosed. A chain is generated that represents a plurality of events in chronological order. The chain comprises a head and one or more nodes. The head and the one or more nodes are stored in a persistent data store as a plurality of key-value pairs. For a request to add data indicative of a new event to the chain, the head is updated in the data store to include the data indicative of the new event.