Deep Pagination System for Distributed Data Retrieval

The patent segments the large data retrieval task into multiple smaller batches using pagination. Instead of retrieving all data objects in a single operation, the system divides the results into manageable pages (e.g., 1000 objects per batch) and retrieves them sequentially. This segmentation reduces the sorting and processing burden on the coordinating node for each individual request, thereby maintaining system performance while still enabling access to large quantities of data objects across multiple paginated requests.

The patent applies preliminary action by having shard nodes pre-sort and prepare their local data objects before returning them to the coordinating node. Each shard node performs initial sorting and filtering of its portion of the data, so that when results are aggregated at the coordinating node, less additional sorting is required. This preliminary processing at the shard level reduces the overall sorting costs and computational burden on the coordinating node, enabling efficient retrieval of large numbers of data objects.

The patent segments the data retrieval into multiple smaller paginated requests rather than attempting to return all results in a single large batch. Each paginated request handles a manageable number of objects (e.g., 1000 per page), which prevents any single request from overwhelming the system and causing failures. This segmentation improves system reliability by ensuring that individual requests remain within safe operational limits, while still enabling efficient retrieval of large datasets through sequential paginated access.

The patent creates a multi-functional system that can handle both small and large dataset retrieval scenarios using the same underlying mechanism. The deep pagination framework maintains the performance benefits of conventional pagination for small requests while extending capability to handle large datasets through automated multi-batch retrieval. The system universally applies pagination logic regardless of dataset size, adapting the number of batches automatically based on the total number of results, thus maintaining performance while increasing adaptability to handle datasets of any scale.