Virtual File Links for Distributed Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, moving data between nodes for load balancing is processor intensive and requires unnecessary storage operations, especially during recovery, as conventional methods involve physically moving data and writing logs across nodes.
Innovation Solution
The implementation of virtual file links allows data to be accessed through a link in a destination virtual file, which is mapped to a chain of pages in a page buffer, enabling data to be provided from the source node without physically moving data, thus reducing the need for explicit synchronization and log operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is physically moved from source node to destination node for load balancing, then data distribution is improved, but processor intensity and storage operations increase
Solution Approach 1:
The patent creates a copy of the data container metadata (including log information) at the destination node while keeping the actual data at the source node. This allows the destination node to have a functional copy that can serve requests without requiring physical data movement, thereby reducing processor intensity and storage operations.
Solution Approach 2:
The patent introduces a metadata container as an intermediary that contains links or references to the actual data container. This intermediary allows the system to track and manage data location without requiring physical movement of the actual data, reducing the computational overhead associated with data migration.
2Productivity
If data containers are moved between nodes, then load balancing is achieved, but additional storage consumption occurs
Solution Approach 1:
The patent creates lightweight copies of metadata containers at destination nodes that contain references to the actual data. This approach achieves load balancing by distributing metadata awareness across nodes without duplicating the actual data, thereby avoiding additional storage consumption.
Solution Approach 2:
The patent separates the data container into two segments: the actual data remaining at the source node and the metadata container (including log information) being distributed to the destination node. This segmentation allows load balancing to be achieved through metadata distribution while the bulk data remains stationary, avoiding additional storage requirements.
3Ease of operation
If conventional data movement operations are performed, then data relocation is achieved, but operation complexity increases
Solution Approach 1:
The patent simplifies data movement operations by copying only the metadata container (which contains links and log information) to the destination node while leaving the actual data at the source. This reduces operation complexity from moving entire data containers to merely copying lightweight metadata structures.
Solution Approach 2:
The patent introduces a metadata container as an intermediary that manages data location and access. This intermediary simplifies the data movement operation by decoupling the logical data structure from its physical location, allowing destination nodes to access data through the metadata container without requiring complex data transfer operations.
Data Source
AI summary
A record within a destination virtual file is generated on a destination node of a distributed data storage system. The record comprises a link directed to a source virtual file stored on a source node. The source virtual file is mapped to a chain of linked pages stored in a page buffer of the distributed data storage system. A request is later received at the destination node to access data defined by the destination virtual file. Data is provided, in response to the request, from the source virtual file stored on the source node using the link. Related apparatus, systems, techniques and articles are also described.