Lock-Free Stateless Paging Mechanism for Distributed Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data stores face bottlenecks when responding to queries with large result sets, leading to resource consumption and potential overload, especially as the data store grows larger.
Innovation Solution
Implementing a paging mechanism that returns subsets of results, using a marker to indicate the extent of the data set, allowing subsequent queries to retrieve disjoint subsets, thereby limiting resource usage and preventing bottlenecks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the data store returns all results for a query with large result sets, then the client receives complete data, but node resources are consumed and bottlenecks occur
Solution Approach 1:
The patent divides large result sets into smaller pages, where each page contains a subset of results. The data store returns one page at a time instead of all results simultaneously, reducing memory consumption and preventing bottlenecks while allowing clients to retrieve complete data through multiple requests.
2Quantity of substance
If the data store processes large result sets, then comprehensive queries are answered, but buffer overflows and data overflows occur
Solution Approach 1:
By segmenting large result sets into manageable pages, the patent prevents buffer overflows and data overflows. Each page fits within available buffer constraints, eliminating the harmful overflow effects while still allowing processing of arbitrarily large data volumes through sequential pagination.
3Loss of information
If the data store handles queries with large result sets, then complete information is provided, but multiple operations cannot interleave
Solution Approach 1:
The patent segments query processing into multiple smaller page-generation operations. This allows the data store to interleave multiple query operations, as each operation only needs to generate one page at a time rather than completing an entire large result set before responding to other requests.
Solution Approach 2:
The patent implements periodic action by returning results in periodic pages rather than as a single continuous operation. Each page represents a periodic delivery of results, allowing the system to pause and handle other operations between pages while maintaining information completeness through sequential delivery.
Data Source
AI summary
Disclosed are a method and apparatus for limiting the number of results returned by a data store in response to a query. Upon receiving an initial query, the data store returns a page of results that includes a subset of the data items within the data store satisfying the conditions of the query. The data store also provides a marker indicating the extent of the set of data items. If a subsequent query that requests additional results which satisfy the same query conditions and that includes the marker is received, the data store returns a page of results that includes a subset of data items that are disjoint from the initial subset, and provides an updated marker which indicates the extent of the union of the initial and subsequent subsets. If still further results are desired from the data store, an additional query containing the updated marker may be submitted.