Partial File Layout Commits for Lower Server Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file layout commit operations in parallel network file systems (pNFS) are inefficient due to smaller file layout sizes and all-or-nothing commit mechanisms, leading to increased server load and decreased performance.
Innovation Solution
Implementing larger file layouts and partial commit operations, where the server maintains an in-memory filesystem block mapping and allows clients to write data directly to storage locations, tracking written and unwritten blocks, and requesting partial commits based on client-generated extent list data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional all-or-nothing commit operations are used, then data consistency is maintained, but the number of commit operations increases and server load increases
Solution Approach 1:
The patent divides the traditional all-or-nothing commit operation into multiple partial commit operations. Each partial commit operation commits only the subset of blocks that have been written, rather than requiring all blocks in the file layout to be written before committing. This segmentation reduces the frequency and size of commit operations, thereby reducing server load while maintaining data consistency for the committed portions.
Solution Approach 2:
The patent implements partial commit operations that commit only the necessary subset of blocks rather than requiring complete file layout commitment. The client device can commit individual extents or groups of extents independently, allowing the system to perform partial actions (committing only written blocks) rather than requiring excessive action (committing all blocks in the layout), thus improving productivity while managing server load.
2Adaptability or versatility
If smaller file layout sizes are used, then client can write data more flexibly, but the number of commit operations increases
Solution Approach 1:
The patent segments the file layout into multiple extents that can be independently committed. The client receives a file layout comprising multiple extents and can commit individual extents or groups of extents based on writing progress. This segmentation maintains the adaptability of smaller layout units while reducing the total number of commit operations by allowing batch commitments of multiple extents in a single partial commit operation.
Solution Approach 2:
The patent allows partial commit operations where the client can commit only the extents that have been written, rather than requiring commitment of the entire file layout. This enables flexible data writing with smaller layout units while reducing the number of commit operations by committing partial portions (written extents) rather than requiring complete layout commitment.
3Reliability
If all blocks must be committed before any data is accessible, then data consistency is ensured, but I/O performance decreases
Solution Approach 1:
The patent segments the commit operation into partial commits that can be performed on individual extents or groups of extents independently. Each partial commit ensures data consistency for the committed portion while allowing other extents to remain in progress. This segmentation enables I/O operations to proceed on committed extents without waiting for all blocks to be committed, thereby improving I/O performance while maintaining reliability for committed data.
Solution Approach 2:
The patent implements partial commit operations that commit only the necessary subset of blocks rather than requiring complete file layout commitment before data accessibility. The client can perform partial commits on written extents, ensuring data consistency for those extents while allowing I/O operations to proceed on other extents, thus improving overall I/O performance without sacrificing reliability of committed portions.
Data Source
AI summary
Partial commits of file layouts can be performed and managed. In response to a file layout request from a client device (CD), a server can communicate, to CD, first extent list data, comprising a first mapping of a group of blocks, in a data store of the server, to which CD is able to write data to the file. CD can write, and track the writing of, data to first blocks, but not second blocks, of the group of blocks. CD can generate second extent list data comprising a second mapping of the first blocks in the data store to which data is written. CD can communicate a partial commit request, comprising the second extent list data, to the server. In response to the partial commit request, the server can commit the second mapping relating to the first blocks to the data store.