DHT Logging Service Skipchain Data Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed applications require a highly available and low-latency distributed data store for logging and retrieving operational data, but existing solutions struggle to ensure durability and high availability of event data.
Innovation Solution
A distributed hash table (DHT) based logging service using a skipchain data structure that organizes data into immutable nodes across distributed partitions, with redundancy and skip nodes for efficient data retrieval and recovery from failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a distributed data store is implemented for logging event data, then availability and durability are improved, but latency increases
Solution Approach 1:
The distributed data store is segmented into multiple partitions distributed across different nodes. Each partition independently stores subsets of the event log, allowing parallel operations and reducing contention. This segmentation enables the system to achieve both high availability through distribution and low latency through localized access to specific partitions.
Solution Approach 2:
The system pre-assigns partitions to specific nodes and pre-establishes replication relationships before data ingestion begins. This preliminary organization of data storage locations and replication paths eliminates the need for complex runtime decision-making, thereby reducing latency while maintaining the availability benefits of distribution.
2Reliability
If data is distributed across multiple nodes for high availability, then reliability is improved, but system complexity increases
Solution Approach 1:
Each node in the distributed system autonomously manages its own partition data, including handling write operations, maintaining local replicas, and performing self-healing when failures occur. This self-service approach eliminates the need for complex centralized coordination mechanisms, thereby reducing overall system complexity while maintaining high availability through distributed autonomy.
Solution Approach 2:
The system implements a simplified replication strategy where each partition is replicated to a predetermined number of nodes regardless of actual failure conditions. This excessive replication approach reduces the complexity of failure detection and recovery logic while ensuring high availability, as nodes can independently serve requests without complex coordination.
3Reliability
If redundancy nodes are added for failure recovery, then durability is improved, but storage overhead increases
Solution Approach 1:
Redundancy is applied locally at the partition level rather than globally across the entire event log. Each partition maintains replicas only on specific designated nodes, allowing the system to optimize storage overhead by concentrating redundancy where it is most needed for durability while avoiding unnecessary replication elsewhere in the system.
Solution Approach 2:
The system implements a retention policy where redundant replica data is discarded when the original data is successfully recovered or confirmed durable. This approach reduces long-term storage overhead while maintaining durability during the critical window when data is being written and replicated, allowing the system to balance durability requirements with storage efficiency.
Data Source
AI summary
A distributed hash table (DHT) based logging service utilizes a DHT to store an event log or another type of data. For example, applications can utilize the logging service to store data that logs aspects of their operation. A key/value storage service can provide access to the DHT. The data stored in the DHT can be organized into a “skipchain” data structure. The data structure includes nodes storing the data that are distributed across partitions of the DHT. Nodes in the data structure can identify “redundancy nodes” and “skip nodes.” When a request is received to add new data to the skipchain data structure, a new node is stored in the DHT that includes the new data. A head node for the data structure is updated in the DHT. The head node can also be utilized to recover from a failure to write an internal node to the data structure.