Data File Clustering with KD-Classifier Trees for Fewer Rewrites
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large data repositories with clustered data files is challenging due to the significant delays and computational overhead caused by rewriting many files when incorporating new or modified data with uniform distribution, especially in cloud storage systems.
Innovation Solution
Implementing a KD-classifier tree that clusters data files based on key-values, allowing efficient data and metadata skipping, and minimizing the number of files rewritten during data modifications by assigning records to nodes with higher granularity, thereby reducing computational burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data files are clustered by key-values to enable efficient read operations, then read operation efficiency is improved, but the complexity of managing data files increases significantly
Solution Approach 1:
The patent introduces a classifier tree as an intermediary data structure that mediates between the clustered data files and the management system. The classifier tree automatically routes data insertion and query operations to the appropriate files based on key-values, eliminating the need for complex manual management while preserving the benefits of clustering.
Solution Approach 2:
The system performs preliminary clustering of data files by key-values before actual data operations. This pre-organization allows read operations to efficiently skip irrelevant files and write operations to automatically determine the correct target files, improving both read efficiency and simplifying management.
2Reliability
If new or modified data with uniform distribution is incorporated into clustered data files, then data completeness is improved, but the number of files that must be rewritten increases significantly
Solution Approach 1:
The patent extracts only the specific data files that contain or should contain the new or modified data based on key-value matching through the classifier tree. This selective approach ensures data completeness while minimizing the number of files that need to be rewritten, as opposed to rewriting all files or using complex distribution algorithms.
3Stability of the object's composition
If data files are rewritten to incorporate new data with uniform distribution, then data distribution uniformity is improved, but the time required to complete transaction operations increases
Solution Approach 1:
The classifier tree is built in advance with knowledge of the data distribution and key-value ranges. This preliminary structure allows the system to quickly determine which files need to be rewritten for uniform distribution, avoiding time-consuming analysis during transaction operations while maintaining both uniformity and speed.
Data Source
AI summary
A data processing service generates a data classifier tree for managing data files of a data table. The data classifier tree may be configured as a KD-classifier tree and includes a plurality of nodes and edges. A node of the data classifier tree may represent a splitting condition with respect to key-values for a respective key. A node of the data classifier tree may be associated with one or more data files assigned to the node. The data files assigned to the node each include a subset of records having key-values that satisfy the conditions represented by the node and parent nodes of the node. The data processing service may efficiently cluster the data in the data table while reducing the number of data files that are rewritten when data is modified or added to the data table.