Virtual File Links for Distributed Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, partitioning large data containers across multiple nodes is processor-intensive and requires moving data and associated logs, leading to performance issues and additional storage needs.
Innovation Solution
The generation of a record with a link to a source virtual file and partition criteria on a destination node allows for efficient data access and partitioning, eliminating the need for explicit data movement and synchronization during recovery, using virtual file links to manage data partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data containers are partitioned across multiple nodes in a distributed data storage system, then memory limitations and performance issues are addressed, but processor intensity and storage consumption increase due to moving data and associated logs
Solution Approach 1:
The patent segments the data container into multiple partitions distributed across different nodes. Each partition is independently managed with its own metadata, allowing parallel processing and reducing the processor intensity required for operations on large datasets. The segmentation enables the system to address memory limitations by distributing data across nodes while maintaining efficient access patterns.
Solution Approach 2:
The patent introduces a namespace service as an intermediary that manages metadata and virtual file links across the distributed system. This intermediary handles the complexity of data location and access coordination, reducing the processor intensity required at individual nodes. The namespace service acts as a mediator that enables efficient data partitioning and access without requiring each node to process all metadata operations.
2Quantity of substance
If data containers are partitioned across multiple nodes, then memory limitations are addressed, but additional storage is consumed for moving data and logs
Solution Approach 1:
The patent uses virtual file links that reference data partitions across nodes rather than physically copying entire data containers. The virtual file system creates lightweight references (links) to the actual data locations, allowing the system to address memory limitations through logical partitioning without consuming additional storage for duplicate data copies. Only metadata about the partitions is stored at each node, not the actual data.
Solution Approach 2:
The patent implements a universal namespace service that handles multiple functions including metadata management, data location resolution, and virtual file link creation. This multi-functional approach eliminates the need for separate storage structures for each function, reducing overall storage consumption while enabling data partitioning across multiple nodes to address memory limitations.
3Adaptability or versatility
If conventional database systems move data and logs during partitioning, then data distribution is achieved, but operation complexity and time increase
Solution Approach 1:
The patent performs preliminary actions by pre-establishing virtual file links and metadata structures in the namespace service before actual data partitioning operations. This preliminary setup enables rapid data distribution across nodes without requiring complex real-time coordination during the partitioning process. The namespace service pre-resolves data locations and creates access paths, reducing the time required for actual data movement and distribution operations.
Data Source
AI summary
A record within a destination virtual file is generated on a destination node of a distributed data storage system. The record comprises (i) a link directed to a partition of a source virtual file stored on a source node and (ii) partition criteria characterizing the partition. The source virtual file is mapped to a chain of linked pages stored in a page buffer of the distributed data storage system and the partitioning criteria is used by at least one of the source node and the destination node to identify data associated with the partition. A request is later received at the destination node to access data defined by the destination virtual file. Data is provided, in response to the request, from the partition of the source virtual file stored on the source node using the link and the partitioning criteria. Related apparatus, systems, techniques and articles are also described.