Large File Truncation via Tree Dereferencing and Rebalancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems are not ideal for latency-sensitive operations, and existing data platforms face challenges in efficiently truncating large files due to the need for linear resource consumption in garbage collection, making on-demand truncation impractical.
Innovation Solution
The data platform performs fast truncation of large files by dereferencing ancestor nodes in a tree data structure and rebalancing the tree data structure according to specific constraints, achieving logarithmic time complexity instead of linear, thereby reducing resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If individual nodes are traversed and updated to truncate an object, then the truncation is performed accurately, but the time complexity is linear O(n)
Solution Approach 1:
The patent segments the tree data structure into subtrees and identifies ancestor nodes that can be dereferenced as a group. Instead of processing each node individually, the system segments the traversal path and dereferences entire subtrees at once, transforming the operation from linear O(n) to logarithmic O(log n) time complexity while maintaining truncation accuracy.
Solution Approach 2:
The patent merges multiple individual node dereferencing operations into a single batch operation. By identifying a set of ancestor nodes and dereferencing them collectively, the system combines what would otherwise be sequential individual operations into a unified logarithmic-time operation, achieving both speedup and accuracy.
2Use of energy by moving object
If background garbage collection is used for truncation, then resource consumption is reduced, but truncation cannot be performed on-demand
Solution Approach 1:
The patent performs preliminary identification of ancestor nodes and determination of the maximum key before executing the truncation. This preliminary action prepares the data structure for efficient on-demand truncation by pre-computing the truncation boundary, enabling the operation to be executed quickly when needed rather than relying on background garbage collection.
Solution Approach 2:
The system implements self-service truncation by directly modifying the tree data structure through ancestor node dereferencing and rebalancing, eliminating the need for external garbage collection processes. The truncation operation serves itself by maintaining the tree's integrity and balance, reducing resource consumption while enabling on-demand execution.
3Loss of time
If the tree data structure is trimmed by dereferencing ancestor nodes, then truncation time is reduced to logarithmic, but the tree structure becomes unbalanced
Solution Approach 1:
The patent changes the structural parameters of the tree data structure by dereferencing ancestor nodes and removing subtrees. This parameter change transforms the tree from a balanced state to an unbalanced state temporarily, achieving logarithmic truncation time. The subsequent rebalancing operation then restores the structural parameters to maintain stability.
Solution Approach 2:
The patent applies dynamic rebalancing to the tree data structure after truncation. The tree structure is allowed to change dynamically from balanced to unbalanced during the truncation operation, then automatically rebalanced to restore stability. This dynamic approach enables the system to temporarily sacrifice balance for speed, then recover balance when needed.
4Quantity of substance
If distributed storage systems are used for backup and archive, then storage scalability is improved, but latency-sensitive operations suffer
Solution Approach 1:
The patent introduces an intermediary tree data structure that sits between the distributed storage system and the user application. This intermediary structure enables efficient on-demand truncation operations by providing a logarithmic-time dereferencing mechanism, bridging the gap between the scalable but slow distributed storage and the latency-sensitive user operations.
Data Source
AI summary
Techniques are described for fast large file truncation. An example method includes receiving, by a data platform, a request to truncate a file, wherein data of the file is stored in a tree data structure including a plurality of nodes with leaf nodes corresponding to data of the file; determining a first node of the plurality of nodes including a plurality of child nodes; determining, based on a maximum key, a subset of the child nodes corresponding to a portion of the data that is to be retained; based on determining not to traverse the first node, updating a second node that is a parent of the first node to include a pointer to each child node of the subset of child nodes; and moving, from a subset of leaf nodes referenced by the subset of child nodes, a leaf node to rebalance the tree data structure.