Dynamic Pagination Tables for Massive API Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional application program interfaces (APIs) face inefficiencies in paginating massive data, leading to increased database burden and transmission time, especially with offset-based and time-based pagination methods, which require additional sorting logic and are prone to data sheet quantity exponential growth, while cursor-based pagination is easily affected by data volume.
Innovation Solution
A data pagination automation method and system that determines a sorting basis using an ordered unique key, sorts and paginates data accordingly, updates a pagination information table dynamically, and collects data using a cursor value to reduce service costs by optimizing pagination logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If offset-based pagination is used, then data can be paginated, but additional sorting logic is required which exponentially increases the quantity of data sheets
Solution Approach 1:
The system pre-generates and stores pagination information in a dedicated pagination information table before actual data queries. This preliminary action includes calculating and storing start/end row numbers, sorting keys, and other pagination metadata, eliminating the need for complex real-time sorting logic during data retrieval operations.
Solution Approach 2:
A pagination information table serves as an intermediary structure between the database and the application layer. This table stores pre-computed pagination metadata that mediates the complex sorting and pagination logic, allowing simple queries to retrieve paginated results without requiring complex sorting operations at query time.
2Ease of operation
If complete data is downloaded for filtering, then filtering can be performed, but the burden of the database and the transmission time is increased
Solution Approach 1:
The invention extracts only the essential pagination information (start row number, end row number, sorting key) from the complete data set and stores it in a separate pagination information table. This extraction allows the system to retrieve only the necessary data subsets for each page without downloading or processing complete data sets, significantly reducing transmission time and database burden.
Solution Approach 2:
The data retrieval process is segmented into two distinct operations: first, querying the pagination information table to obtain pagination metadata; second, using this metadata to query only the specific data subset needed for the current page. This segmentation eliminates the need to download and filter complete data sets, reducing transmission time while maintaining filtering capability.
3Ease of operation
If cursor-based pagination is used, then pagination can be implemented, but it is easily affected by the volume of data
Solution Approach 1:
The system dynamically adjusts pagination parameters based on data volume characteristics. The pagination information table stores adaptive parameters such as start/end row numbers and sorting keys that are recalculated based on the actual data volume and distribution, allowing the pagination mechanism to adapt to varying data volumes without performance degradation.
Data Source
AI summary
A data pagination automation method, for processing a plurality of data items includes determining a sorting basis for sorting the plurality of data items according to an ordered unique key of the plurality of data items; sorting and paginating the plurality of data items according to the sorting basis; and updating a pagination information table of the plurality of data items and collecting corresponding data according to the ordered unique key and at least a filtering condition; wherein the pagination information table is updated with to a partial dynamic update method.