Commit-Time Ordered Message Queue for Repeatable Reads

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing message queue systems fail to maintain data consistency and repeatable reads due to lack of transactional dependency ordering, leading to data dependency violations and non-repeatable reads when handling concurrent enqueuing of dependent messages.

Innovation Solution

Implementing a commit-time ordered message queue that associates a unique system commit time with each transactional message group, using a high watermark to ensure only fully determined messages are visible, and employing an index-only table for efficient storage and retrieval of commit system change numbers to enforce transactional dependencies and guarantee repeatable reads.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If enqueue-time ordering is used for message queue, then messages can be dequeued quickly without waiting for transaction commits, but data dependency ordering is violated and read consistency cannot be guaranteed

Engineering Contradiction:

Improvemessage dequeue speedVSAvoiddata consistency

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system performs preliminary actions by capturing messages at enqueue time and staging them in a buffer, then applies commit-time ordering later during the dequeue phase. This allows the system to prepare messages in advance while ensuring they are released in the correct transactional order, resolving the contradiction between early availability and consistent ordering.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A buffer or staging area acts as an intermediary between the enqueue operation and the dequeue operation. Messages are captured in this intermediate structure with their commit timestamps, allowing the system to decouple the timing of message capture from message release, ensuring that messages are dequeued in commit order while being captured at enqueue time.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If commit-time ordering is enforced for all messages, then data dependency ordering is guaranteed, but messages cannot be dequeued until all transactions commit

Engineering Contradiction:

Improvetransactional dependency orderingVSAvoidmessage availability delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system applies commit-time ordering selectively rather than universally. High-priority or time-sensitive messages can be dequeued earlier with appropriate ordering guarantees, while maintaining strict commit-time ordering for messages where transactional dependency is critical. This partial application of ordering rules reduces unnecessary delays.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The message queue system dynamically adjusts its ordering behavior based on message priorities, transaction characteristics, and system state. Rather than enforcing a static commit-time ordering rule for all messages, the system can adaptively determine the appropriate dequeue timing and ordering strategy for each message, balancing consistency requirements with availability needs.

Inventive Principle:

Principle #15Dynamics

3Productivity

If multiple subscribers access the message queue concurrently, then system throughput increases, but read consistency and repeatable reads become difficult to guarantee

Engineering Contradiction:

Improvesystem throughputVSAvoidread consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The message queue is segmented into multiple ordered streams or partitions, each maintaining its own commit-time ordering. Multiple subscribers can concurrently access different segments or apply consistent snapshots of the ordered stream, allowing parallel processing while preserving read consistency within each segment. This segmentation enables both high throughput and consistent reads.

Inventive Principle:

Principle #1Segmentation

4Adaptability or versatility

If arbitrary read and dequeue patterns are allowed, then subscriber flexibility increases, but maintaining commit-time ordering becomes complex

Engineering Contradiction:

Improvesubscriber flexibilityVSAvoidordering management complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system creates and maintains copies of the commit-time ordered message stream at different points in time or for different subscribers. Rather than requiring all subscribers to read from a single complex ordering structure, each subscriber can access a simplified copy or view of the ordered stream that meets its specific needs, reducing the complexity of managing arbitrary access patterns while preserving ordering guarantees.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS7680793B2Commit-time ordered message queue supporting arbitrary read and dequeue patterns from multiple subscribers

Publication Date: 2010.03.16 ORACLE INT CORP

AI summary

Data consistency in the context of information sharing requires maintenance of dependencies among information being shared. Transactional dependency ordering is implemented in a database system message queue, by associating a unique system commit time with each transactional message group. Read consistency is implemented in such a queue by allowing only messages with a fully determined order to be visible. A fully determined order is implemented through use of a high watermark, which guarantees that future transactions, for which messages are entering the queue, have commit times that are greater than the current high watermark. Therefore, only messages below the current high watermark are visible and can be dequeued, with no chance of other new messages enqueuing below the current high watermark.