Federated Key-Value Store for Cost-Effective Logging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale online services face challenges in implementing cost-effective and performant logging solutions due to the high volume of log information, with on-premises solutions incurring maintenance and reliability costs and cloud-based solutions being costly for storing large data volumes.
Innovation Solution
A federated computing system utilizing a multi-level index with a Key-Value Storage System for low-latency data access and an Object Storage System for high-capacity data storage, where key-value items are organized into 'runs' based on time, allowing for efficient insertion and retrieval of log data, and offloading older data to reduce storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If on-premises logging service with many computing nodes is implemented, then logging capacity and reliability are improved, but maintenance overhead and implementation costs increase
Solution Approach 1:
The logging service is segmented into multiple independent computing nodes forming a distributed cluster, where each node handles a portion of the logging workload. This segmentation allows the system to scale reliability by adding more nodes while maintaining independent failure domains, reducing the impact of single node failures.
Solution Approach 2:
The computing nodes in the distributed cluster are designed with multi-functionality, serving both as logging collection points and as part of the analytics processing infrastructure. This universal design reduces overall system complexity by eliminating dedicated single-purpose components.
2Reliability
If on-premises logging service is implemented, then data control and reliability are improved, but implementation and maintenance expenses increase
Solution Approach 1:
The distributed logging cluster implements self-service capabilities through automated node discovery, dynamic load balancing, and self-healing mechanisms. When nodes are added or removed from the cluster, the system automatically reconfigures itself without requiring manual intervention, reducing maintenance expenses while maintaining data reliability.
3Ease of manufacture
If cloud-based data store is used, then maintenance costs are reduced, but storage costs for large data volumes increase
Solution Approach 1:
The system dynamically changes storage parameters by implementing a tiered storage architecture that automatically migrates log data between hot storage (for recent logs requiring frequent access) and cold storage (for historical logs). This parameter change optimizes the balance between maintenance effort and storage cost based on data access patterns and age.
4Productivity
If distributed data storage system is implemented, then scalability and cost-effectiveness are improved, but system complexity increases
Solution Approach 1:
A centralized coordinator service acts as an intermediary between the distributed storage nodes and the logging applications. This mediator handles complex operations such as data routing, consistency management, and failure recovery, allowing individual nodes to remain simple while achieving overall system scalability and robustness.
Data Source
AI summary
One variation of a system for implementing a key-value data store includes one or more processors, storage media and instructions stored in the storage media which, when executed by the system cause the system to: receive a request store a particular key-value item; request a first networked distributed data storage system to store the particular key-value item; based on a determination that a set of one or more offload criteria is satisfied: retrieve a first set of key-value items from the first networked distributed data storage system, and request a second networked distributed data storage system to store the first set of key-value items in a first set of one or more data objects. The first networked distributed data storage system can have a lower data write latency and a higher data storage cost than the second networked distributed data storage system.