Machine-Learned Clustering Key Selection for Faster Data File Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing services face inefficiencies in clustering data files due to manual selection of non-optimal clustering keys, leading to excessive reading of data files and wastage of computing resources.
Innovation Solution
A machine-learned clustering key selection model is trained to predict optimal clustering keys using a large language model (LLM), which is then used to construct a data classifier tree for efficient data file management and reduce unnecessary data reading.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual selection of clustering keys is performed, then user control over data organization is maintained, but sub-optimal clustering keys are selected leading to increased time and computing resources
Solution Approach 1:
The system enables self-service by allowing the data processing service to automatically select clustering keys using machine-learned models, eliminating the need for manual user selection while optimizing data processing efficiency through algorithmic key selection based on data patterns and access patterns
Solution Approach 2:
The manual mechanical process of user selection is replaced with an automated machine-learned model that analyzes data characteristics and automatically determines optimal clustering keys, substituting human decision-making with computational intelligence to improve processing efficiency
2Device complexity
If manual selection of clustering keys is performed, then simplicity of the system is maintained, but the data processing service has to read more data files than necessary
Solution Approach 1:
The machine-learned model performs preliminary analysis of data characteristics and access patterns to pre-determine optimal clustering keys before data processing operations begin, enabling the system to organize data in advance for optimal retrieval performance and reduce read operation time
Solution Approach 2:
The system changes the parameter of key selection from static manual choice to dynamic machine-learned optimization, allowing the clustering keys to be automatically adjusted based on data patterns and access patterns to minimize the number of data files that need to be read
3Ease of manufacture
If manual selection of clustering keys is performed, then ease of implementation is maintained, but computing resources are expended inefficiently
Solution Approach 1:
The manual implementation process is replaced with automated machine-learned model deployment that efficiently selects clustering keys, substituting human effort with computational algorithms that optimize resource utilization and reduce wasted computing resources during data processing operations
Data Source
AI summary
The disclosed configurations provide a method (and/or a computer-readable medium or system) for determining, from a table schema describing keys of a data table, one or more clustering keys that can be used to cluster data files of a data table. The method includes generating features for the data table, generating tokens from the features, generating a prediction for each token by applying to the token a machine-learned transformer model trained to predict a likelihood that the key associated with the token is a clustering key for the data table, determining clustering keys based on the predictions, and clustering data records of the data table into data files based on key-values for the clustering keys.