Data Store Controller with Buffer for Real-Time Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data networks face challenges in efficiently capturing and managing large data flows and querying data in real-time, particularly when data is unavailable, due to issues with data store read rates and buffer utilization.
Innovation Solution
A system comprising an input to receive data, a writer to store data in a data store, a controller to regulate read and write rates, and query functions that access data from the data store or shared memory, with prioritization of writing over reading and indexing for efficient data retrieval and querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the controller increases the read rate from the data store to support real-time queries, then query responsiveness is improved, but data loss occurs when the write rate exceeds the read rate
Solution Approach 1:
The system pre-loads data into a buffer before it is needed for queries. The controller proactively reads data from the data store and stores it in buffer memory in advance, so that when queries arrive, the data is already available in the buffer, eliminating the need to increase read rates and preventing data loss.
2Reliability
If the system prioritizes writing data to the data store over reading, then data capture completeness is improved, but query response time increases due to slower read rates
Solution Approach 1:
Data is pre-loaded into the buffer during periods when query demand is low, in advance of when it will be needed. This preliminary action ensures that when queries arrive, the data is already available in the buffer, maintaining fast query response times while allowing the system to prioritize writing to the data store.
Solution Approach 2:
The buffer acts as an intermediary between the data store and the query function. It decouples the write operations from read operations, allowing the system to prioritize writing to the data store without impacting query response times, as queries can be served from the buffer independently.
3Productivity
If the controller maintains a high read rate to support multiple simultaneous queries, then query throughput is improved, but data is lost when the write rate fluctuates below the read rate
Solution Approach 1:
The buffer serves as an intermediary that absorbs fluctuations in write rates. When the write rate is high, excess data is stored in the buffer. When the write rate drops, the buffer continues to supply data to queries, maintaining high query throughput without causing data loss, as the buffer compensates for the fluctuation.
Solution Approach 2:
The buffer provides a cushion of pre-stored data that protects against write rate fluctuations. This beforehand cushioning ensures that even when the write rate drops below the read rate, queries can continue to be served from the buffer without causing data loss or reducing query throughput.
4Loss of time
If the system uses a large buffer to store pre-loaded data, then query response time is improved, but memory usage increases
Solution Approach 1:
The buffer size and data retention policy are dynamic rather than static. The controller adjusts the buffer contents based on query patterns and data freshness requirements, keeping only the most recently written or most frequently queried data in the buffer. This dynamic management maintains fast query response times while optimizing memory usage.
Solution Approach 2:
The system changes parameters such as buffer size, data retention period, and pre-loading thresholds based on workload conditions. During high query demand, the buffer is maintained at optimal size; during low demand, buffer contents are adjusted or cleared, allowing query response time to be improved when needed while minimizing memory usage during normal operation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system is arranged to receive data which is written to a data store by a writer. A controller is able to read data from the data store. That controller is able to control the rate at which data is read from the data store with respect to the rate at which data is written to the data store. A query function receives a stream of said data substantially in real time and when said stream of data is unavailable in real time, the query function is able to subsequently obtain that unavailable data from said data store.