Virtual File Partitioning for Parallel Database Restoration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems experience significant latency in processing very large files due to the lack of native support for parallelized processing of file sections during operations like backup, restoration, and duplication.
Innovation Solution
Distribute virtual partitions of large files across multiple nodes of a data management system for parallel processing, enabling parallelized handling of file sections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data management operations process large files sequentially on single nodes, then system complexity remains low, but latency increases significantly
Solution Approach 1:
The patent divides large files into multiple sections and creates virtual partitions across different nodes. Each node processes specific sections in parallel, transforming a single sequential operation into multiple concurrent operations. This segmentation enables the system to process different portions of large files simultaneously, significantly reducing overall latency while maintaining manageable complexity through structured partitioning.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by distributing file sections across multiple nodes in a networked system. Instead of single-node sequential processing, the system leverages the dimensional expansion of having multiple computing nodes work simultaneously on different file sections, thereby reducing latency without proportionally increasing complexity.
2Productivity
If data management systems support parallel processing of file sections, then productivity increases, but device complexity increases
Solution Approach 1:
By segmenting files into virtual partitions distributed across nodes, the system enables parallel processing of multiple file sections simultaneously. This segmentation strategy increases productivity by allowing concurrent operations on different file portions while keeping complexity manageable through standardized partitioning mechanisms.
Solution Approach 2:
The patent creates a universal virtual partitioning mechanism that can handle various data management operations (backup, restore, duplication) across different file types and sizes. This multi-functional approach increases productivity for diverse operations while reducing complexity by providing a unified parallel processing framework rather than separate solutions for each operation type.
3Productivity
If virtual partitions of large files are distributed across multiple nodes, then parallel processing efficiency improves, but ease of operation deteriorates
Solution Approach 1:
The patent introduces a file descriptor mechanism that acts as an intermediary between users and the distributed virtual partition system. Users interact with files through standard file descriptors without needing to understand the underlying partitioning or node distribution. This intermediary layer maintains ease of operation by hiding complexity while enabling parallel processing efficiency through the virtual partition architecture.
Data Source
AI summary
Methods, systems, and devices for data management are described. Point-in-time data that includes one or more files may be generated from data management information stored at a data management system having multiple nodes. One or more sets of virtual partitions may be created for at least one file of the one or more files. One or more external file descriptors associated with respective locations of the at least one file and one or more sets of internal file descriptors associated with respective external file descriptors and locations of corresponding sets of virtual partitions may be generated in response to a request. One or more subsequent requests to read the at least one file may be routed to the nodes based on the one or more sets of internal file descriptors. Based on the routing, the respective portions of the one or more files may be output in parallel.