Selective Data Retrieval for Pivot Table Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard pivoting techniques fail to handle large datasets due to memory limitations and often include irrelevant data, leading to inefficient data processing and presentation.
Innovation Solution
The system selectively retrieves relevant data from a relational database using Structured Query Language (SQL) to create a limited dataset, which can be used to generate a pivot table on a client device, by filtering or calculating only the necessary rows based on user-defined criteria, such as grouping, partition, and ordering columns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If standard pivoting techniques are used to handle large datasets, then complete data is processed, but memory limitations prevent successful execution
Solution Approach 1:
The patent segments the large dataset into smaller manageable chunks by selectively retrieving only the necessary rows required for the pivot table. Instead of loading the entire dataset into memory, the system divides the data retrieval process into targeted queries that fetch only relevant portions, allowing pivot operations to complete successfully within memory constraints.
Solution Approach 2:
The patent extracts only the essential data elements needed for pivot table generation from the larger dataset. By using SQL queries with specific filtering criteria (grouping, partition, ordering columns), the system extracts precisely the rows required, excluding all irrelevant data, thereby reducing memory requirements while maintaining pivot table functionality.
2Loss of information
If all data from the dataset is included in the pivot table, then complete information is provided, but irrelevant data reduces processing efficiency
Solution Approach 1:
The patent extracts only the relevant rows needed for the pivot table by applying specific SQL query criteria based on the grouping, partition, and ordering columns. This extraction process eliminates irrelevant data from the dataset, ensuring that only necessary information is processed and included in the final pivot table, thereby maintaining data completeness while improving processing efficiency.
Solution Approach 2:
The patent applies local quality by treating different portions of the dataset differently based on their relevance to the pivot table requirements. Rather than uniformly processing all data, the system selectively retrieves only those rows that meet specific criteria, giving different 'quality' of inclusion to different data portions - relevant rows are included while irrelevant ones are excluded.
3Ease of operation
If the entire dataset is loaded into memory for pivoting, then all data can be manipulated, but memory constraints are exceeded
Solution Approach 1:
The patent extracts only the necessary rows from the dataset using targeted SQL queries that filter based on pivot table requirements. This extraction allows the system to work with a reduced dataset that fits within memory constraints while maintaining full data manipulation capability for the pivot operation, as all necessary data for grouping, partitioning, and ordering is included.
Solution Approach 2:
The patent performs preliminary data filtering and selection before the pivot operation by executing SQL queries that pre-identify and retrieve only the rows needed for the pivot table. This preliminary action ensures that when data is loaded into memory, it is already in the correct format and quantity, eliminating the need to load and then filter the entire dataset, thus preventing memory overflow.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure provides inclusive and exclusive approaches, including computer-implemented methods for selectively retrieving data from an initial dataset in a relational database to provide a limited dataset for incorporation into a pivot table. Also provided are related systems and non-transitory computer-readable recording media.