Dynamic Range Partitioning for Unknown Data Sizes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in dynamically performing range partitioning during execution time due to unknown data sizes and constraints related to file size and number of rows, which hinders efficient data querying and storage.
Innovation Solution
A method and system for execution-time dynamic range partitioning that determines split points and generates partitioning quantiles based on user data size, total rows, target file size, and target number of rows per file, allowing for dynamic range partitioning of user data into files sorted by a clustering key, while accommodating storage constraints such as maximum number of partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is partitioned using static partitioning parameters determined before execution, then the partitioning structure can be pre-established, but the system cannot adapt to unknown data sizes and storage constraints at execution time
Solution Approach 1:
The system performs preliminary actions by establishing the partitioning framework and quantile computation logic before execution, but defers the actual partition point determination to execution time when data characteristics are known. This allows the system to prepare the partitioning structure in advance while adapting to actual data sizes and storage constraints during execution.
Solution Approach 2:
The partitioning parameters are made dynamic by computing split points at execution time based on actual data characteristics and storage constraints. The system transitions from static pre-defined partitions to dynamic execution-time partitions, allowing adaptation to unknown data sizes and varying storage requirements.
2Productivity
If the number of partitions is increased to improve query performance on large datasets, then data can be better distributed across storage, but the complexity of managing and querying partitioned data increases
Solution Approach 1:
The system changes the parameter of partition count from a fixed static value to a dynamic value determined at execution time. By computing the optimal number of partitions based on actual data size and storage constraints, the system optimizes query performance without manually managing partition complexity.
Solution Approach 2:
The system uses feedback from storage constraints and data characteristics to dynamically adjust the number and distribution of partitions. By monitoring actual data sizes and storage capacity, the system automatically optimizes partitioning to improve query performance while managing complexity.
3Productivity
If data is stored in a clustered structure sorted by clustering key, then related data can be co-located for efficient querying, but the inability to dynamically determine partitioning parameters hinders optimal data organization
Solution Approach 1:
The system performs preliminary sorting by clustering key within each partition while maintaining the flexibility to determine optimal partition boundaries at execution time. This preliminary organization enables efficient data retrieval while adapting to dynamic partitioning requirements.
Solution Approach 2:
The clustering structure is made dynamic by allowing partition boundaries to be determined at execution time based on actual data characteristics. This enables optimal co-location of related data while maintaining the ability to adapt to varying data sizes and query patterns.
Data Source
AI summary
An example method includes receiving a data load request requesting loading and partitioning of an unknown quantity of user data for storage at a data storage system. The user data including a partitioning key; a total data size of the user data; a plurality of rows, each row of the plurality of rows associated with a value defined by the partitioning key; and one or more columns. The method also includes identifying one or more storage constraints for the data storage system. The method further includes, after receiving the user data, determining a plurality of partitioning quantiles defining respective ranges of values of the partitioning key based on the user data and the one or more storage constraints for the data storage system; and range partitioning each row of the user data into files based on the value associated with the row defined by the partitioning key, and the respective ranges of the values of the partitioning key defined by the plurality of partitioning quantiles.