CQRS Data Store Architecture for Low-Latency Reliable Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems of records struggle with high query traffic volumes, leading to operational issues in terms of cost, scalability, and reliability, necessitating a mechanism for low latency data storage that ensures rapid accessibility while maintaining data integrity and consistency.
Innovation Solution
Implementing a non-relational database with a Command Query Responsibility Segregation (CQRS) architecture, where the query part is horizontally scaled to handle read operations within less than one second or 100 milliseconds, and incorporating a data reconciliation framework to track and remedy event count mismatches, along with a store-and-forward function for system resilience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional mainframe-based systems of records are used to ensure reliability of command traffic, then data integrity and reliability are improved, but query response time and scalability deteriorate
Solution Approach 1:
The system divides the database into separate command and query parts, allowing write operations to be processed independently from read operations. This segmentation enables the query part to be optimized for fast read access while the command part maintains data integrity through write operations, resolving the contradiction between reliability and speed.
Solution Approach 2:
The system introduces an event stream as an intermediary between the command part and query part. Events are published to an event stream when written, allowing the query part to consume and process these events asynchronously. This intermediary mechanism enables fast query responses without blocking write operations, thus improving both reliability and speed.
2Adaptability or versatility
If query traffic volume is increased to meet real-time data access demands, then system versatility and adaptability are improved, but operational issues in cost and scalability worsen
Solution Approach 1:
The system implements dynamic scaling of the query part based on read workload demands. The query part can be horizontally scaled to handle increased query traffic without affecting the command part. This dynamic adaptation allows the system to meet real-time data access demands while maintaining scalability and controlling costs.
Solution Approach 2:
The system pre-processes and stores data in an optimized format for queries in the query part, while events are published asynchronously to an event stream. This preliminary preparation allows fast query responses without increasing system complexity, as the heavy lifting is done in advance during event processing.
3Productivity
If horizontally scaled query part is implemented to reduce query response time, then speed and productivity are improved, but data consistency and reliability may worsen
Solution Approach 1:
The system implements a feedback mechanism where the command part publishes events to an event stream after successful writes. The query part subscribes to this event stream and consumes events to update its data. This feedback loop ensures that queries always see the most recent data, maintaining consistency while allowing horizontal scaling for high throughput.
Solution Approach 2:
The event stream provides continuous updates of data changes from the command part to the query part. This continuous flow of events ensures that the horizontally scaled query part always has access to up-to-date information, maintaining data consistency while enabling high query throughput through parallel processing.
Data Source
AI summary
Various methods and processes, apparatuses or systems, and media for operating a low latency data store that is designed to provide rapid accessibility to data while ensuring data integrity, consistency, and reliability are disclosed. The method includes: receiving a set of data; storing the set of data in a storage medium; receiving, from a user, a query that relates to the set of data; and providing a subset of the set of data to the user in response to the query. The storage medium includes a non-relational database that implements a Command Query Responsibility Segregation (CQRS) architecture within which a command part performs write operations and a query part performs read operations. The query part is horizontally scaled such that the provision of the subset of the set of data is performable within less than one second of the receiving of the query.