Cloud Database Table Writer with Sparse Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud-based database systems are inefficient and costly due to the need to retrieve large datasets to query a portion of the data, leading to slow performance and high expenses.
Innovation Solution
The method involves a table writer program that sorts and divides data into datapages, creates sparse indices, and writes them to cloud storage with a footer containing schema, primary-key, and entitlements, optimizing data retrieval by allowing precise querying and access control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in cloud storage without structured organization, then storage simplicity is maintained, but query retrieval efficiency deteriorates and costs increase
Solution Approach 1:
The patent segments data into rows and columns, organizes them into tables with primary keys, and divides tables into datapages. This segmentation enables efficient querying by allowing the system to retrieve only specific datapages containing relevant data, rather than scanning entire datasets, thus improving query retrieval efficiency while maintaining manageable structural complexity
Solution Approach 2:
The patent performs preliminary actions by pre-sorting data by primary key, pre-dividing data into datapages, and pre-creating sparse indices during the data writing phase. These preliminary organizational actions enable fast query execution later, as the data is already structured and indexed, resolving the contradiction between retrieval efficiency and storage complexity
2Loss of time
If large datasets are retrieved to query a portion of data, then query completeness is ensured, but retrieval time and cost increase
Solution Approach 1:
The patent extracts only the necessary data by using sparse indices to identify and retrieve specific datapages containing the queried information, rather than retrieving entire datasets. The sparse indices store metadata about data distribution and indexing information, enabling the system to extract and transfer only the relevant datapages, significantly reducing retrieval time and data volume
Solution Approach 2:
The patent introduces sparse indices as an intermediary structure between the data and the query. The sparse indices act as a mediator that maps query conditions to specific datapages, allowing the system to navigate and retrieve only the necessary data portions without scanning the entire dataset, thus reducing both retrieval time and data volume
3Measurement precision
If data is written to cloud storage without indexing, then storage operation simplicity is maintained, but query precision and access control deteriorate
Solution Approach 1:
The patent applies local quality by creating sparse indices at the datapage level rather than requiring a complete index of the entire dataset. Each datapage has its own sparse index containing only the necessary indexing information for that page, enabling precise queries while reducing the overall complexity compared to a full dataset index
Solution Approach 2:
The patent uses partial action by creating sparse indices that contain only the necessary indexing information for each datapage, rather than creating complete indices for the entire dataset. This partial indexing approach provides sufficient query precision for most use cases while significantly reducing the complexity and storage overhead of the indexing structure
Data Source
AI summary
A method for writing data to a cloud-based database for optimized cloud-based database query execution may include a table writer computer program executed by a local electronic device: (1) receiving table data comprising rows and columns, a schema for the table, and a primary-key for the table; (2) creating a footer for the table, the footer comprising the schema and the primary-key; (3) sorting the rows of the table data using the primary-key; (4) dividing the sorted columns of the table data into a plurality of datapages based on a datapage size; (5) identifying sparse indices for each datapage, wherein the sparse indices comprising a minimum value, a maximum value, a starting row index, an ending row index for each datapage; (6) writing each datapage to cloud storage; (7) inserting the sparse indices into the footer for the table data; and (8) writing the footer to cloud storage.