Branch-Based Data Storage with Snapshot Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies due to the copy-on-write method, which requires reading and rewriting existing data to incorporate new data, leading to increased overhead and reduced storage efficiency.
Innovation Solution
Implementing a data storage method that utilizes a target storage system with trunk and experimental branches, where new data is stored on an experimental branch without affecting the trunk branch, allowing for isolated data processing and reduced overhead by using snapshots to indicate storage paths and manage data storage efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If copy-on-write method is used to add new data to storage system, then data consistency is maintained, but data storage efficiency deteriorates due to increased reading and writing overhead
Solution Approach 1:
The storage system is segmented into multiple independent branches (trunk branch and experimental branches). Each branch maintains its own data snapshot, allowing new data to be written to experimental branches without affecting the trunk branch. This segmentation eliminates the need to read and rewrite entire datasets when adding new data, thus improving storage efficiency while maintaining data consistency through snapshot isolation.
Solution Approach 2:
Instead of using copy-on-write that copies data upon modification, this patent uses snapshot copying that creates a point-in-time copy of the entire branch. When new data needs to be added, a new snapshot is created for the experimental branch, and data is written directly to this snapshot without reading or rewriting existing data in the trunk branch. This reduces I/O overhead significantly.
2Ease of operation
If all data files are merged into a single storage structure, then data management is simplified, but processing overhead increases due to the need to read and merge data from all branches
Solution Approach 1:
The patent segments data files into different branches (trunk and experimental) that are logically separated but managed within a unified storage system. Each branch has its own snapshot and data files, allowing selective access without reading unrelated data. This segmentation reduces processing overhead by enabling targeted data access while maintaining manageable data organization through branch-specific metadata.
Solution Approach 2:
The patent introduces branch-specific metadata and snapshot mechanisms as intermediaries between the storage system and data access operations. This intermediary layer allows the system to manage multiple branches independently while providing a unified interface for data operations. The metadata tracks which data files belong to which branches, enabling efficient selective access without scanning or merging all data files.
Data Source
Figure 1
Figure 2~3
Figure 4~6
AI summary
Embodiments of the present disclosure provide a data storage method and apparatus, an electronic device, and a storage medium, and the data storage method includes: acquiring, in response to a file storage request for a first target data file, the first target data file to be stored in a target storage system, in which the target storage system is configured with table metadata, the table metadata is used to indicate storage branch information corresponding to the target storage system, the storage branch information includes a trunk branch and at least one experimental branch, the trunk branch is used to mount a data file that has been used for model training and whose training result meets a preset requirement, and the experimental branch is used to mount a data file that has not been used for model training; determining a first target experimental branch corresponding to the file storage request based on the table metadata, and newly creating a first target snapshot corresponding to the first target experimental branch; and storing, based on the first target snapshot, the first target data file to a corresponding storage location in the target storage system. The embodiments of the present disclosure are capable of improving data storage efficiency.