Two-Stage Message Logging for Multi-Tenant Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tenant database systems face challenges in reducing latency and increasing storage efficiency for message data logging, often resulting in bottlenecks and delays due to simultaneous write operations on a single storage server.
Innovation Solution
A two-stage message logging process is implemented, where message data is initially stored in job-specific, write-optimized staging tables across multiple servers to distribute write operations, and then periodically moved to the corresponding tenant space, utilizing partitioning and sharding to avoid hotspots and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If message data is logged to a single storage server in multi-tenant database systems, then data consistency is maintained, but latency increases and storage efficiency decreases due to bottlenecks from simultaneous write operations
Solution Approach 1:
The patent segments the monolithic storage server into multiple storage servers (first storage server, second storage server, etc.) and distributes message data across them. Each storage server handles a portion of the logging workload, eliminating the bottleneck of a single server and reducing latency while maintaining data consistency through coordinated access.
Solution Approach 2:
The patent introduces a message server as an intermediary between the logging system and multiple storage servers. The message server receives message data, determines which storage servers should receive it, and coordinates the distribution. This intermediary manages the complexity of multi-server coordination while enabling parallel writes that reduce latency.
2Device complexity
If message data is stored in a single storage server, then system complexity is low, but storage efficiency decreases due to bottlenecks and hotspots
Solution Approach 1:
The storage system is segmented into multiple storage servers that can process write operations in parallel. This segmentation increases storage efficiency by distributing the workload and eliminating hotspots, while the modular architecture manages complexity through standardized interfaces and protocols.
Solution Approach 2:
The patent transitions from a single-dimension storage architecture (one server) to a multi-dimensional distributed architecture. Message data is routed across multiple storage servers based on various criteria, adding spatial distribution as a new dimension that improves storage efficiency without proportionally increasing system complexity.
3Ease of operation
If write operations are concentrated on a single storage server, then data access is simplified, but latency increases due to storage bottlenecks
Solution Approach 1:
Write operations are segmented and distributed across multiple storage servers. Each server handles a subset of the write operations, increasing the overall speed of data ingestion. The segmentation allows parallel processing while maintaining manageable access patterns through standardized protocols.
Data Source
AI summary
Systems, methods, and computer-readable media are provided for logging message data with reduced the latency and decreased overhead in multi-tenant systems. A message server may implement a two-stage process for logging the message data. In the first stage, the message server may create job-specific and write-optimized staging tables for each message processing job, and may store the message data for each job in a corresponding staging table. This may be done as the message server is building and sending the messages for each job. Additionally, the message server may partition or shard the staging tables so that the write operations are spread to multiple storage disks to avoid creating a storage server hotspot or bottleneck. In the second stage, the message server may move the accumulated message data from the individual staging tables to a corresponding tenant space at regular intervals. Other embodiments may be described and/or claimed.