Partial Order Message Queue for Asset Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Asset management systems face challenges with concurrent updating and message ordering issues, leading to data corruption and invalid states due to the limitations of pessimistic and optimistic locking schemes, and the rapid distribution of messages by conventional message queues, which result in low data quality and accuracy.
Innovation Solution
The system employs a partial order message queue that groups messages based on asset relationships, sorts them by generation time, and processes them in order, while holding out-of-order messages until earlier ones are received, ensuring correct ordering and reducing locking conflicts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional message queues distribute messages rapidly to multiple threads, then throughput is improved, but message ordering is lost and data accuracy deteriorates
Solution Approach 1:
The message queue is segmented into multiple partial order queues, each responsible for maintaining order for a specific subset of messages or asset types. This allows different segments to be processed concurrently while preserving ordering within each segment, resolving the contradiction between throughput and ordering accuracy.
Solution Approach 2:
A mediator component is introduced that receives out-of-order messages, buffers them temporarily, and redistributes them in the correct sequence to the appropriate processing threads. This intermediary layer preserves message ordering without blocking the overall high throughput of the system.
2Reliability
If pessimistic locking is used to prevent concurrent updating, then data integrity is improved, but system throughput deteriorates due to transaction queuing
Solution Approach 1:
The locking mechanism is segmented so that different threads can hold locks on different assets simultaneously. By dividing the asset space into partitions and allowing concurrent locking within each partition, the system maintains data integrity while enabling parallel processing across multiple assets, thus preserving throughput.
Solution Approach 2:
Locking is applied locally to specific assets rather than globally to all assets. Each thread acquires locks only on the assets it needs to modify, allowing other threads to concurrently access and modify different assets. This local quality approach maintains data integrity for each asset while enabling high overall system throughput.
3Productivity
If optimistic locking is used to maximize throughput, then system throughput is improved, but data reliability deteriorates due to transaction failures
Solution Approach 1:
The system implements feedback mechanisms that monitor lock contention and transaction failure rates. When conflicts are detected, the system dynamically adjusts locking strategies, message routing, or thread scheduling to prevent conflicts before they cause failures. This feedback loop maintains high throughput while improving transaction success rates by avoiding failed optimistic locking scenarios.
4Productivity
If messages are processed concurrently by multiple threads, then throughput is improved, but concurrent updating problems arise leading to data corruption
Solution Approach 1:
The message processing system is segmented into multiple independent processing channels, each dedicated to specific asset types or message categories. Within each channel, messages are processed sequentially to prevent concurrent updating conflicts, while multiple channels operate in parallel to maintain high overall throughput. This segmentation resolves the contradiction between concurrent processing and data consistency.
Solution Approach 2:
The system performs preliminary actions by acquiring necessary locks and validating message sequences before processing begins. By preparing the processing environment in advance and ensuring proper ordering and locking are in place before concurrent execution, the system enables safe parallel processing that maintains data consistency while achieving high throughput.
Data Source
AI summary
A system and method provides efficient message processing of assets. The system and method places all messages in a queue as they are received from an information feed and groups the messages according to the entities with which they are associated. The messages within each group are sorted according to the time the message was generated by the information feed, and then are processed in the order in which they were generated.