Data Management System for Efficient File Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems are inefficient and resource-intensive when restoring individual files from large data backups, as they often require scanning and transferring entire disk snapshots.
Innovation Solution
A data management system that uses concurrent readers and writers to efficiently transfer and manage individual files within virtual machines, allowing for concurrent reading and writing of data chunks and periodic checkpointing to track file transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire disk snapshot is restored to retrieve individual files, then data recovery is achieved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent divides the disk snapshot into individual file components, allowing the system to retrieve only the specific files needed rather than restoring the entire snapshot. This segmentation enables selective file recovery by identifying and extracting individual files from the snapshot data structure.
Solution Approach 2:
The patent extracts only the necessary individual files from the disk snapshot instead of restoring the complete snapshot. The system identifies target files within the snapshot and extracts them selectively, eliminating the need to process and restore unnecessary data portions.
2Reliability
If the entire disk snapshot is restored to retrieve individual files, then data recovery is achieved, but network resources are excessively consumed
Solution Approach 1:
The patent segments the data transfer process by transmitting only the specific individual files that need to be recovered, rather than transferring the entire disk snapshot over the network. This reduces network bandwidth consumption and transfer time.
Solution Approach 2:
The system extracts and transfers only the necessary file data from the snapshot, eliminating network transmission of unnecessary snapshot metadata and other file contents. This selective extraction significantly reduces network resource utilization.
3Reliability
If the entire disk snapshot is restored to retrieve individual files, then data recovery is achieved, but the process is excessively time-consuming
Solution Approach 1:
The patent segments the restoration process into individual file operations, allowing parallel processing and selective recovery of files. This enables the system to recover only the required files without waiting for the entire snapshot restoration to complete.
Solution Approach 2:
The system performs preliminary identification of target files within the snapshot before initiating the recovery process. By pre-locating and marking the specific files to be recovered, the system avoids scanning and processing the entire snapshot, significantly reducing restoration time.
4Productivity
If individual files are retrieved from the data store, then resource efficiency is improved, but file transfer tracking becomes complex
Solution Approach 1:
The patent implements a feedback mechanism that tracks the transfer status of individual files from the data store to the target system. The system monitors which files have been successfully transferred and maintains this state information, providing visibility into the recovery progress without requiring complex manual tracking.
Data Source
AI summary
A data management system manages concurrent readers and writers for large file scans. The data management system may read a plurality of data chunks of the file starting from different offsets and generate a bounded number of read requests, which causes a data chuck identifiable by a data offset to be loaded into a data buffer. The system may queue the loaded data chunks for generating write requests to release the loaded data chunks. One or more write requests are generated responsive to one or more data chunks being associated with a consecutive order of data offsets being successfully loaded to data buffers. The system may write data chunks released from the buffer-rounded reading stage to the data storage in a checkpointed writing stage. The checkpointed writing stage creates a checkpoint based on the data offset of the data chunks that have been completely transferred to the data storage.