Low-Latency Demultiplexer for Ordered Multi-Sink Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Propagating ordered data from a database to multiple sinks with varying consumption rates is challenging, as existing solutions either require separate queries for each sink, leading to high latency, or limit all sinks to the slowest rate, increasing latency for faster sinks.
Innovation Solution
A database client mediates data propagation by buffering writes in volatile memory, using confirmation indicators to ensure data is sent to sinks in the order of commitment, allowing each sink to consume data at its own rate without additional storage or database reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate independent database queries are used for each sink, then each sink can consume data at its own rate, but the system requires multiple database connections and queries leading to increased complexity and latency
Solution Approach 1:
The patent combines multiple sink data streams into a single demultiplexer that receives one database connection and distributes data to multiple sinks. The demultiplexer maintains separate buffers for each sink while using a single database interface, merging the database connection layer while preserving individual sink consumption rates.
Solution Approach 2:
The demultiplexer acts as an intermediary between the database and multiple sinks. It receives data from the database once and redistributes it to multiple sinks with different consumption rates, eliminating the need for each sink to have its own direct database connection.
2Stability of the object's composition
If all sinks are limited to the rate of the slowest sink, then data order is maintained, but latency increases for all faster sinks
Solution Approach 1:
The patent segments the data stream into separate buffered streams for each sink. Each sink has its own buffer that can be filled independently, allowing faster sinks to receive data immediately while slower sinks receive data at their pace. The segmentation occurs at the buffer level, not at the database query level.
Solution Approach 2:
The system dynamically adjusts data flow to each sink based on individual consumption rates. The demultiplexer monitors buffer levels and data readiness for each sink independently, allowing the system to adapt to varying sink speeds without imposing a uniform rate limit on all sinks.
3Productivity
If data is buffered in volatile memory for ordering, then data can be propagated to multiple sinks efficiently, but additional memory resources are required
Solution Approach 1:
The volatile memory buffers serve multiple functions: they maintain data order, decouple the database write rate from sink consumption rates, and enable efficient data distribution to multiple sinks. This multi-functionality reduces the need for separate memory allocations for each purpose.
Data Source
AI summary
A computer-implemented method includes receiving a series of writes. Each write in the series of writes is requesting respective data to be stored at a database. For each respective write in the series of writes, the method includes: storing the respective data in a buffer associated with one or more data streaming applications; transmitting the respective data to the database; receiving, from the database, an order indicator indicating an order the respective data of the respective write was committed to the database relative to other writes in the series of writes; and in response to receiving order indicator, sending the respective data of the respective write from the buffer to each data streaming application of the one or more data streaming applications based on the order indicator indicating the order the respective data was committed to the database relative to other writes in the series of writes.