Distributed Storage Data Append via Atomic Description Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud storage systems, users face challenges with writing large data files due to network limitations and concurrent write issues, particularly in distributed systems, where append write mode is not effectively supported, leading to data integrity concerns.
Innovation Solution
The method involves acquiring and storing user data in a newly allocated data storage block, generating and updating data start-end information, and using atomic operations to persist the description file, allowing for append operations without overwriting existing data, and managing data storage blocks to maintain data integrity and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If append write mode is supported in distributed storage system, then user flexibility and data management capability are improved, but data integrity and consistency may be compromised due to concurrent write issues
Solution Approach 1:
The patent segments the data storage into multiple blocks and uses a description file to manage metadata about each block. When appending data, the system creates new storage blocks and updates the description file to point to these blocks, rather than modifying existing blocks in place. This segmentation approach allows concurrent writes without corrupting existing data while maintaining append flexibility.
Solution Approach 2:
The patent performs preliminary actions by pre-allocating new storage blocks before appending data. The system reserves space for incoming data and only links it to the description file after allocation is complete. This preliminary allocation ensures that concurrent write operations do not interfere with each other, as each operation works on its own allocated blocks rather than sharing existing blocks.
2Productivity
If multiple write operations are performed to complete data file writing, then network limitations are addressed, but concurrent write conflicts and data integrity issues arise
Solution Approach 1:
The patent uses copying by creating new storage blocks for appended data rather than modifying existing blocks. The description file is updated to copy references to these new blocks. This copying mechanism allows multiple concurrent write operations to proceed independently on different blocks without causing conflicts, as each operation copies data to its own allocated space rather than sharing mutable space.
Solution Approach 2:
The description file acts as an intermediary that manages the relationship between data blocks and user operations. Instead of direct concurrent modifications to the same data structure, the description file mediates by maintaining a unified view of all storage blocks and their relationships. This intermediary structure allows multiple writers to operate concurrently while the description file ensures consistency by tracking all modifications centrally.
3Quantity of substance
If existing data is overwritten during append operations, then data storage efficiency is improved, but data integrity is damaged
Solution Approach 1:
The patent segments data into distinct storage blocks with clear boundaries. Existing data remains in its original blocks while new data is placed in newly allocated blocks. The description file maintains separate references to both old and new blocks, ensuring that appending operations never overwrite existing data. This segmentation provides both storage efficiency through block-level management and data integrity through isolation of data regions.
Data Source
AI summary
The present application provides methods and devices for writing data and acquiring data in a distributed storage system. According to an exemplary method, in a process of writing data, a data file is not overwritten but is instead updated by making a copy in a newly allocated storage block and an updated description file is persisted by using an atomic operation. This way, in a process of appending data, it can be ensured that a data file is not damaged, and the readability and consistency of data are maintained, facilitating the use by a user. Further, writing data according to some embodiments of the present application supports a write in which a user specifies a write position of to-be-written data, and also supports a write in which the user does not specify a write position of to-be-written data, thereby improving the flexibility of writing data.