Event Sourcing Datastore With Caching Layer For Read Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing event sourcing systems face challenges in providing both strong consistency and high scalability, failing to meet the requirements of business-to-consumer applications.
Innovation Solution
The implementation of an event sourcing datastore that uses a web tier and a streaming tier, with an event processor enforcing event ordering, idempotence, and constraints, and an aggregate processor that asynchronously aggregates events into an aggregate log, stored in a caching layer for high throughput and low latency, utilizing event sourcing templates to ensure consistent and performant event handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If event sourcing datastores are built on relational databases using ACID transactions, then strong consistency is provided, but scalability is limited to one or a few nodes
Solution Approach 1:
The system divides the event sourcing datastore into multiple independent nodes that can operate in parallel. Each node maintains a portion of the event logs and can process events independently, enabling horizontal scalability while maintaining consistency through coordinated operations across segments.
Solution Approach 2:
An event log service acts as an intermediary layer between event producers and event consumers. This mediator manages event distribution, ordering, and consistency across multiple database nodes, allowing the system to scale horizontally while preserving strong consistency guarantees through centralized coordination of the intermediary layer.
2Productivity
If event sourcing datastores are built on NoSQL using sharding, then linear scalability is provided, but strong consistency guarantees are lost
Solution Approach 1:
The system segments the event log into distributed partitions across multiple NoSQL database nodes, enabling linear scalability through horizontal sharding. Each partition maintains local consistency while the overall system achieves strong consistency through coordinated event ordering and distribution mechanisms managed by the event log service.
Solution Approach 2:
The event log service provides multiple functions including event storage, ordering, distribution, and consistency management across distributed nodes. This multi-functional intermediary enables the system to achieve both scalability through NoSQL sharding and strong consistency through centralized coordination, resolving the trade-off between these two requirements.
3Measurement precision
If aggregate states are computed in real-time from event logs, then data accuracy is maintained, but read throughput decreases and latency increases
Solution Approach 1:
The system pre-computes aggregate states from event logs and stores them in a caching layer before they are needed. Event subscribers can then retrieve pre-computed aggregates directly from cache without performing real-time computations, maintaining data accuracy while significantly improving read throughput and reducing latency.
Solution Approach 2:
The system replaces the mechanical computation process with a caching mechanism. Instead of computationally processing event logs in real-time when data is requested, the system substitutes this with efficient cache lookups of pre-computed aggregate states, dramatically improving performance while maintaining accuracy through periodic cache refreshes from the event log.
4Productivity
If aggregate states are cached for fast retrieval, then read throughput increases and latency decreases, but memory and computational resources are consumed
Solution Approach 1:
The system implements a caching layer that stores pre-computed aggregate states locally for fast retrieval. By caching data at the point of use rather than computing it centrally each time, the system improves read throughput and reduces latency while managing memory resources through localized, targeted caching of frequently accessed aggregate states.
Data Source
AI summary
Systems, methods, and computer-readable media for event sourcing datastores are provided. Events are sent to an event sourcing datastore (ESDS) from an external service, and the events are appended to an event log. The ESDS maintains the event log in one table and the events are automatically aggregated via streaming. The ESDS is linearly scalable, and enforces uniqueness, consistency, and user-defined constraints when appending events to the event log. The aggregate state is computed asynchronously and transparently cached, which provides increased read throughput and reduced latency. Other embodiments may be described and/or claimed.