KD-Epsilon Tree Data Clustering for Cloud Storage Efficiency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Managing a significant number of clustered data files is challenging, especially when new or modified data has a uniform distribution with respect to key-values, leading to significant delays in transaction operations on cloud storage systems.

Innovation Solution

The use of a KD-epsilon tree data structure to efficiently cluster data files, allowing for data skipping and reducing the number of files that need to be rewritten during data modifications, by utilizing nodes that represent splitting conditions with respect to key-values and storing metadata and changes in dedicated buffers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If data files are clustered by key-values to improve read operation efficiency, then read performance is improved, but the number of files that need to be rewritten during data modifications increases significantly

Engineering Contradiction:

Improveread operation speedVSAvoiddata modification efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The patent segments the data modification process by introducing a separate metadata file that tracks key-value distributions across clustered data files. This allows the system to maintain clustering benefits for reads while avoiding the need to rewrite entire data files during modifications, as the metadata can be updated independently to reflect new data distributions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a metadata file as an intermediary between the data files and the clustering logic. This metadata file stores information about key-value distributions and enables the system to determine which data files need to be accessed or rewritten during modifications, thereby reducing the overall rewriting burden while maintaining clustering efficiency for reads.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If many data files are rewritten to incorporate new or modified data with uniform distribution, then data consistency is maintained, but transaction operation delay increases

Engineering Contradiction:

Improvedata consistencyVSAvoidtransaction operation delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by pre-computing and storing metadata about key-value distributions in the metadata file before data modifications occur. This allows the system to quickly determine which data files need to be rewritten during transactions, maintaining data consistency while minimizing the time required for modification operations by avoiding full scans or rewrites of unrelated files.

Inventive Principle:

Principle #10Preliminary action

3Productivity

If data is clustered to enable data skipping during reads, then read efficiency is improved, but the complexity of managing clustered data files increases

Engineering Contradiction:

Improveread operation efficiencyVSAvoiddata file management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a metadata file as an intermediary that simplifies the management of clustered data files. This metadata file stores information about key-value distributions and enables the system to perform data skipping during reads without requiring complex management logic for the clustered files themselves. The metadata acts as a lookup table that reduces the complexity of managing clustered data while maintaining read efficiency.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Stability of the object's composition

If uniform distribution data is written to clustered files, then all files may need rewriting to maintain clustering, but this results in significant I/O overhead

Engineering Contradiction:

Improveclustering structure stabilityVSAvoidI/O overhead

Core Design Contradiction:

Stability of the object's compositionVSLoss of energy

Solution Approach 1:

The patent extracts the clustering management information from the data files themselves and stores it separately in a metadata file. This extraction allows the system to maintain clustering structure stability by updating only the metadata when uniform distribution data is written, rather than rewriting entire data files. The metadata file captures the distribution patterns, enabling the system to avoid unnecessary I/O operations on the actual data files while preserving clustering integrity.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS12332862B2Data file clustering with KD-epsilon trees

Publication Date: 2025.06.17 DATABRICKS INC

AI summary

A data tree for managing data files of a data table and performing one or more transaction operations to the data table is described. The data tree is configured as a KD-epsilon tree and includes a plurality of nodes and edges. A node of the data tree may represent a splitting condition with respect to key-values for a respective key. A leaf node of the data tree may correspond to a data file for a data table that includes a subset of records having key-values that satisfy the condition for the node and conditions associated with parent nodes of the node. A parent node may correspond to a file including a buffer that stores changes to data files reachable by this parent node, and also includes dedicated storage to pointers of the child nodes. By using the data tree, the data processing system may efficiently cluster the data in the data table while reducing the number of data files that are rewritten.