Distributed Evaluation List Updating via Event Stream Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional evaluation list updating mechanisms struggle to scale with high traffic volumes and efficiently manage updates in applications, particularly when using traditional SQL-based database systems, which can only support a single writable master instance, leading to performance issues under heavy loads.
Innovation Solution
A distributed evaluation list updating system with a decoupled design that aggregates writes into an event stream for batch processing against multiple master databases, enabling high-velocity queries and out-of-band updates, allowing for incremental or full replacements of list entries, and supporting both database and event stream lookups based on latency requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a traditional SQL-based database system is used to store evaluation lists, then data consistency and structured access are improved, but the system cannot scale to high traffic volumes without specialized caching strategies
Solution Approach 1:
The system segments the monolithic database write operation into multiple independent event stream processors that can operate in parallel. Each processor handles a portion of the update workload, allowing the system to scale horizontally by adding more processors without increasing the burden on a single database instance.
Solution Approach 2:
The system performs preliminary actions by pre-computing and caching evaluation list data in multiple redundant locations before it is needed. This allows read operations to be served from cache rather than requiring real-time database access, thereby improving throughput while maintaining data consistency.
2Reliability
If a single writable master database instance is used, then data integrity is maintained, but the system struggles to update lists that are actively used by an application under high traffic
Solution Approach 1:
The system introduces an event stream as an intermediary layer between the database and application. This mediator captures write operations, queues them for batch processing, and coordinates updates across multiple database instances, thereby maintaining data integrity while enabling parallel processing of updates.
Solution Approach 2:
The system implements periodic batch processing of evaluation list updates instead of immediate individual updates. By aggregating multiple update operations into periodic batches, the system reduces the frequency of write operations to the master database, thereby maintaining integrity while improving overall update throughput.
3Productivity
If specialized caching strategies are implemented to handle high traffic volumes, then request processing speed is improved, but the system complexity increases and requires one-off solutions
Solution Approach 1:
The system implements a multi-functional event stream processing platform that serves multiple purposes: it acts as a message queue for coordinating updates, a cache for storing evaluation data, and a computation engine for processing batch operations. This universal platform handles high traffic volumes without requiring separate specialized caching infrastructures.
Solution Approach 2:
The system merges previously separate functions (caching, update coordination, and batch processing) into a unified event stream processing architecture. By combining these functions into a single system, the patent reduces overall system complexity while maintaining the ability to handle high traffic volumes.
Data Source
AI summary
Methods and systems for providing distributed evaluation list updating. One system includes a data processing server. The data processing server receiving a request event associated with an update to an active evaluation list version. The active evaluation list version is actively accessible for request validation. The data processing server also generates an updated evaluation list version according to the update associated with the request event. After generating the updated evaluation list version, the data processing server replaces the active evaluation list version with the updated evaluation list version in storage, where, in response to storing the updated evaluation list version, the updated evaluation list version is actively accessible for request validation.