Network Controller Erasure Coding With Version-Checked Data Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, the use of CPU for erasure code encoding and decoding in clients leads to increased overheads, becoming a system bottleneck and affecting data processing efficiency.
Innovation Solution
Offload CPU-intensive erasure code encoding and decoding tasks to a network controller, utilizing it to generate target data blocks with same version information at both ends, reducing CPU load and ensuring data validity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the CPU performs erasure code encoding and decoding, then data processing can be completed, but CPU overheads increase and the CPU becomes a system bottleneck
Solution Approach 1:
The patent extracts the erasure code encoding and decoding functions from the CPU and relocates them to the network controller. This separation removes the computationally intensive tasks from the CPU, reducing its overhead and preventing it from becoming a system bottleneck, while the network controller handles these specialized operations.
Solution Approach 2:
The network controller acts as an intermediary between the CPU and the storage system. It receives data from the CPU, performs the computationally intensive erasure code encoding, and sends the encoded data blocks to storage nodes. During reading, it retrieves data blocks, performs decoding, and returns the original data to the CPU, thereby mediating the data processing workflow and offloading computational burden.
2Reliability
If erasure code encoding is performed to ensure data reliability, then data can be protected against loss, but computing overhead increases
Solution Approach 1:
The patent extracts the computationally intensive erasure code encoding operation from the CPU and places it in the network controller. This maintains the reliability benefits of erasure coding while reducing the computing overhead on the CPU, as the network controller is better suited for handling these specialized encoding operations.
3Productivity
If the network controller performs erasure code encoding and writes target data blocks, then CPU consumption is reduced, but the system must ensure data validity through version information
Solution Approach 1:
The patent applies preliminary action by embedding version information at both ends of target data blocks during the encoding process in the network controller. This pre-established versioning mechanism enables automatic data validity verification during reading operations, ensuring data integrity without adding complex validation logic at other system components.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
This application discloses a data writing method, a data reading method, an apparatus, a device, a system, and a medium. A network controller performs erasure code encoding on original data, and writes a third quantity of target data blocks in a plurality of obtained target data blocks into a storage node. The network controller reads a first quantity of target data blocks in the third quantity of target data blocks from the storage node, and decodes the read target data blocks. The plurality of target data blocks include a first quantity of original data blocks and a second quantity of check data blocks, two ends of the target data block include same version information, and the third quantity is greater than the first quantity. In this application, computing overheads originally belonging to a central processing unit can be offloaded to the network controller, thereby reducing consumption of the central processing unit, improving data processing efficiency, and ensuring data validity by using the version information included at two ends of the target data block.