Document Store Update Segmentation for Priority Sync Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document databases experience significant processing bottlenecks due to the multiplicative effort required for data updates, especially when replicating data from relational databases, leading to long queues and delayed data reflection, particularly exacerbated by bulk data uploads which inhibit the application of low-volume UI interactions.
Innovation Solution
Separate processing of updates based on the source and object type, allocating more computing resources to priority object types, using a three-service architecture where updates from client UI interactions and bulk uploads are handled differently, with a dedicated service for priority object types receiving additional resources to expedite updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data updates are replicated from relational database to document database, then fast fuzzy searching capability is improved, but processing bottlenecks and long update queues are worsened
Solution Approach 1:
The patent segments the update processing system into multiple independent services (first service for low-volume updates, second service for high-volume updates, third service for priority object types) that operate in parallel. This segmentation allows the system to handle different types of updates simultaneously without mutual interference, resolving the bottleneck caused by sequential processing in traditional document database replication.
Solution Approach 2:
The patent implements dynamic resource allocation where computing resources are allocated differently based on update characteristics. The third service receives additional computing resources specifically for priority object types, while other services handle standard updates. This dynamic allocation optimizes processing speed for critical updates without permanently over-provisioning resources for all update types.
2Quantity of substance
If bulk data uploads are processed in document database, then data completeness is improved, but update application for other operations is inhibited
Solution Approach 1:
The patent creates a dedicated second service that segments high-volume bulk upload processing from the main update queue. This service专门 handles bulk data uploads independently, preventing them from blocking low-volume UI interaction updates. The segmentation ensures that bulk operations do not inhibit the speed of other update applications.
Solution Approach 2:
The patent introduces an event streaming platform as an intermediary between the relational database and document database. This intermediary decouples the replication process, allowing bulk uploads to be processed asynchronously without directly blocking other update operations. The event streaming platform buffers and manages the flow of updates, preventing bottlenecks.
3Speed
If additional computing resources are allocated to expedite priority updates, then update speed for priority objects is improved, but system cost is worsened
Solution Approach 1:
The patent applies local quality by allocating additional computing resources specifically to the third service that handles priority object types, rather than uniformly increasing resources across all services. This targeted allocation improves priority update speed while minimizing overall resource consumption and cost. Non-priority updates continue to use standard resource allocation.
Data Source
AI summary
Systems and methods include a first service to receive events from a first event log, a second service to receive events from a second event log, and a third service to receive events from both event logs. The first event log receives low event volumes and the second event log receives high event volumes. The first service and the second service update a document database based on events associated with non-priority data objects and the third service updates the document database based on events associated with priority data objects. More computing resources are employed to execute the third service than to execute the first service or the second service.