Decoupled Operation Log Service for Key-Value Database Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing key-value database systems face performance degradation and resource consumption due to the tight coupling of core services and operation log management services, leading to increased disk IO overhead and reduced efficiency.
Innovation Solution
Implementing a decoupled operation log management service that operates separately from the core service, allowing for independent resource management and offloading of log management tasks, thereby enabling the core service to focus on business logic and reducing resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the core service and operation log management service are tightly coupled, then resource sharing and integration are improved, but performance degradation and resource consumption increase
Solution Approach 1:
The patent segments the database system into two independent services: the core service and the operation log management service. This segmentation allows each service to run on separate processors or processor cores, preventing resource contention and enabling the core service to focus on business logic while the log management service handles logging operations independently.
Solution Approach 2:
The operation log management service is extracted from the core service, removing the logging functionality as a separate, independent service. This extraction eliminates the resource overhead that logging operations impose on the core service, allowing the core service to operate with reduced resource consumption and improved performance.
2Device complexity
If the operation log management service is tightly coupled with the core service, then service integration is improved, but CPU time and memory consumption increase
Solution Approach 1:
The patent segments the database system into two independent services: the core service and the operation log management service. This segmentation allows each service to run on separate processors or processor cores, preventing resource contention and enabling the core service to focus on business logic while the log management service handles logging operations independently.
Solution Approach 2:
The operation log management service is designed to operate autonomously, managing its own resources and operations without requiring continuous involvement of the core service. This self-service capability allows the log management service to handle logging tasks independently, reducing the CPU time and memory consumption of the core service.
3Productivity
If the operation log management service handles all log operations, then log management capability is improved, but disk IO overhead increases
Solution Approach 1:
The patent implements a buffering mechanism where operation logs are temporarily stored in memory before being written to disk. This preliminary action in memory allows multiple log operations to be batched and written to disk in fewer, more efficient I/O operations, reducing the overall disk IO overhead while maintaining complete log management capability.
Data Source
AI summary
Disclosed herein are methods of managing an operation log using an operation log management service that is decoupled from the core service of a key-value database. The decoupled operation log management service receives operation data from a key-value database, processes the operation data thereby generating operation information, and records the operation data and the operation information to the operation log. The decoupled operation log management service allows the core service to focus on core service logic. Also disclosed are methods of concurrently managing operation data using a plurality of threads at an operation log management service. The concurrent data management method resolves data collisions using a hash table. Non-transitory, computer-readable storage media comprising computer-executable instructions, cause a processor, processing unit, or circuit to perform the methods disclosed herein. Systems comprising one or more processors are also capable of performing the methods disclosed herein.