Non-persistent Message Queue Spilling for Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database management systems face performance and memory usage issues due to the acknowledgment mechanism in non-persistent message queues, which leads to increased memory usage and inefficient propagation of messages between source and destination DBMSs.
Innovation Solution
Implementing a spilling strategy that moves non-persistent messages from a destination non-persistent message queue to a persistent message queue based on parameters like sparseness factor, number of messages, and memory resources, with placeholders for spilled messages characterized as consumed to optimize performance and memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an acknowledgment mechanism is utilized to synchronize the propagation of non-persistent messages, then message propagation reliability is improved, but memory usage increases and performance deteriorates
Solution Approach 1:
The patent changes the state of message storage by spilling non-persistent messages from volatile memory (non-persistent message queue) to persistent storage (persistent message queue) based on parameters such as sparseness factor, number of messages, and memory resources. This parameter change resolves the contradiction by maintaining reliability through persistent storage while reducing memory usage in the non-persistent queue.
Solution Approach 2:
The patent extracts spilled messages from the non-persistent message queue and stores them separately in the persistent message queue. By taking out these messages, the system reduces memory usage in the non-persistent queue while maintaining the ability to deliver messages reliably through the persistent storage mechanism.
2Reliability
If an acknowledgment mechanism is utilized to synchronize the propagation of non-persistent messages, then message propagation reliability is improved, but propagation efficiency worsens
Solution Approach 1:
The system dynamically changes the storage parameter of messages based on queue conditions (sparseness factor, message count, memory resources). When messages are spilled to persistent storage, the acknowledgment mechanism operates more efficiently because the non-persistent queue maintains fewer messages, reducing synchronization overhead while preserving reliability.
Solution Approach 2:
The persistent message queue acts as an intermediary for spilled messages. Messages are transferred from the non-persistent queue to the persistent queue, which then serves as a mediator for reliable delivery. This intermediary mechanism improves propagation efficiency by separating the high-speed non-persistent path from the reliable persistent path.
3Productivity
If placeholders for spilled messages are characterized as consumed, then acknowledgment handling efficiency is improved, but message delivery accuracy may worsen
Solution Approach 1:
The patent creates placeholders as copies of spilled messages in the non-persistent queue. These placeholders are characterized as consumed to improve acknowledgment handling efficiency. The actual message content is stored in the persistent queue, ensuring delivery accuracy is maintained through the persistent storage mechanism while the placeholders enable efficient acknowledgment processing.
Solution Approach 2:
Placeholders serve as intermediaries between the non-persistent queue and the acknowledgment mechanism. By characterizing placeholders as consumed, the system improves acknowledgment handling efficiency without affecting actual message delivery accuracy, since the real messages remain in persistent storage until successfully delivered.
Data Source
AI summary
A method of optimizing propagation of non-persistent messages from a source database management system (DBMS) to a destination database management system (DBMS) is described and provided. The method comprises receiving one or more nonpersistent messages from a source non-persistent message queue of the source DBMS at a destination non-persistent message queue of the destination DBMS. The nonpersistent messages are consumed from the destination non-persistent message queue such that consumed non-persistent messages are removed from the destination non-persistent message queue. Further, at least a portion of the non-persistent messages are spilled from the destination non-persistent message queue to a destination persistent message queue of the destination DBMS based on at least one parameter selected from a group consisting of a sparseness factor, number of nonpersistent messages in the destination non-persistent message queue, and amount of memory resources utilized by the destination non-persistent message queue. Moreover, to the source DBMS is sent an acknowledgment including a watermark indication of consumed non-persistent messages from the destination non-persistent message queue. Spilled non-persistent messages are characterized as consumed.