Data processing method and apparatus based on erasure coding, and storage medium

Abstract

The present application relates to the field of cloud computing. Disclosed are a data processing method and apparatus based on erasure coding, and a storage medium. The method comprises: when target data to be sent is detected at a client, segmenting the target data according to a preset data packet size, so as to obtain a plurality of data packets; among the plurality of data packets, if the data volume of a target data packet is less than the preset data packet size, generating check data on the basis of the target data packet and an erasure coding algorithm, and determining a padding data record of the target data packet, wherein the padding data record records position offset information of padding data required in the target data packet; and sending the target data, the check data, and the padding data record to a plurality of data nodes for storage. The present application solves the technical problem of the redundant occupation of a storage space being increased due to the necessity to insert padding data when processing unaligned data during data writing and recovery processes using erasure coding technology.

Description

Data processing method, apparatus and storage medium based on erasure coding

[0001] This application claims priority to Chinese Patent Application No. 202510072146.5, filed on January 16, 2025, entitled “Data Processing Method, Apparatus and Storage Medium Based on Erasure Coding”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing, and more specifically, to a data processing method, apparatus, and storage medium based on erasure coding. Background Technology

[0003] With the rapid development of cloud computing and big data, the storage capacity of distributed storage systems is growing exponentially, posing challenges to the storage and management of massive amounts of data. Meanwhile, given the inherent risk of device failure in distributed architectures, ensuring high data reliability and low storage costs has become a core consideration in the design of distributed storage systems. Erasure coding technology has emerged to address this need. It cleverly balances the relationship between data redundancy and storage efficiency. Through data sharding and the generation of verification information, it not only significantly improves the fault tolerance of data in multi-node environments but also significantly reduces the overall overhead of distributed storage systems, becoming a key technology in the field of storage technology.

[0004] However, traditional erasure coding file writing mechanisms reveal certain limitations when faced with random user write requests. Erasure coding algorithms require strict data alignment for efficient calculation of verification information. However, user write requests often fail to match the data alignment standards, leading to the distributed storage system needing to insert a large amount of padding data to reach the required block size when processing misaligned data. This padding not only consumes valuable storage resources and introduces unnecessary redundancy, but also exacerbates the complexity of storage space management, especially under frequent write operations, making it difficult to effectively manage and reduce this redundant storage.

[0005] There is currently no effective solution to the problem that in the data writing and recovery process using erasure coding technology, padding data needs to be inserted when processing unaligned data, which leads to increased redundant storage space usage. Summary of the Invention

[0006] This application provides a data processing method, apparatus, and storage medium based on erasure coding, to at least solve the technical problem that when processing unaligned data during data writing and recovery using erasure coding technology, padding data needs to be inserted, leading to increased redundant storage space usage.

[0007] According to one aspect of the embodiments of this application, a data processing method based on erasure coding is provided, comprising: when it is detected that there is target data to be sent on the client, dividing the target data into multiple data packets according to a preset data packet size; if, among the multiple data packets, there is a target data packet whose data volume is less than the preset data packet size, calculating and generating verification data based on the target data packet and the erasure coding algorithm, and determining a padding data record for the target data packet, wherein the padding data record records the position offset information of the padding data to be filled in the target data packet; and sending the target data, verification data, and padding data record to multiple data nodes for storage.

[0008] Furthermore, the calculation and generation of verification data based on the target data packet and the erasure coding algorithm includes: padding the target data packet with padding data; and calculating and generating verification data using the target data and padding data according to the erasure coding algorithm.

[0009] According to one aspect of the embodiments of this application, a data processing method based on erasure coding is provided, comprising: receiving target data, verification data and padding data records sent above; generating padding data required for the target data packet to complete the target data packet according to the position offset information in the padding data records; and performing a target operation based on the target data, the padding data and the verification data.

[0010] Furthermore, the position offset information includes: position offset and length information. The target operation performed based on the target data, padding data, and checksum data includes: if the target operation is data recovery, the lost data portion is recovered using the target data and checksum data through an erasure coding algorithm to generate a complete data packet; if the target operation is re-encoding, the target data and padding data are treated as a complete data packet, and new checksum data is recalculated using the position offset and length information of the target data and padding data, along with the checksum data, according to the erasure coding algorithm to update the checksum information; if the target operation is data reading, the target data and padding data are combined to generate a complete data packet to meet the data format requirements of the read request.

[0011] Furthermore, if the padding data record includes a target padding method, generating the padding data required to complete the target data packet based on the position offset information in the padding data record includes: generating the padding data required to complete the target data packet based on the position offset, length information, and target padding method in the padding data record.

[0012] Furthermore, the padding data records are stored on the data nodes in the form of metadata, which includes at least: the position offset of the padding data, length information, and the identifier of the target data packet.

[0013] According to another aspect of the embodiments of this application, a data processing apparatus based on erasure coding is also provided, comprising: a segmentation unit, configured to segment the target data according to a preset data packet size to obtain multiple data packets when the presence of target data to be sent on a client is detected; a determination unit, configured to calculate and generate verification data based on the target data packet and the erasure coding algorithm, and determine the padding data record of the target data packet when the data volume of the target data packet is less than the preset data packet size, wherein the padding data record records the position offset information of the padding data to be filled in the target data packet; and a sending unit, configured to send the target data, verification data and padding data record to multiple data nodes for storage.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes a data processing method based on erasure coding during runtime.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform an erasure coding-based data processing method.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements a data processing method based on erasure coding.

[0017] In this embodiment, when target data to be sent is detected on the client side, the target data is divided into multiple data packets according to a preset data packet size. If the data volume of any target data packet is smaller than the preset data packet size, verification data is calculated and generated based on the target data packet and the erasure coding algorithm. A padding data record for the target data packet is then determined, where the padding data record records the position offset information of the required padding data in the target data packet. The target data, verification data, and padding data record are sent to multiple data nodes for storage. This solves the technical problem of increased redundant storage space due to the need to insert padding data when processing non-aligned data during data writing and recovery using erasure coding technology. In this application, sending the target data, verification data, and padding data record to multiple data nodes for storage avoids sending the target data packet after padding, thereby reducing storage space usage, improving storage efficiency, and minimizing system storage overhead. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of this application;

[0020] Figure 2 is a flowchart of a data processing method based on erasure coding according to Embodiment 1 of this application;

[0021] Figure 3 is a schematic diagram of an optional erasure coding-based data processing method provided according to Embodiment 1 of this application;

[0022] Figure 4 is a flowchart of a data processing method based on erasure coding according to Embodiment 2 of this application;

[0023] Figure 5 is a flowchart of performing a target operation based on target data, fill data and verification data according to Embodiment 2 of this application;

[0024] Figure 6 is a schematic diagram of a data processing device based on erasure coding according to Embodiment 3 of this application;

[0025] Figure 7 is a schematic diagram of a data processing device based on erasure coding according to Embodiment 4 of this application;

[0026] Figure 8 is a structural block diagram of an electronic device provided according to Embodiment 5 of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0030] Erasure coding is a fault-tolerant coding technique. Its basic principle is to divide the stored data into fragments, and then use a certain verification calculation method to generate k+m fragments of data from k original data. The original data can be restored from any k+m fragments. In this way, even if some data is lost, the distributed storage system can still recover the original data.

[0031] A packet is a data fragment distributed across a single computer device, consisting of multiple consecutive data blocks.

[0032] A packet group is the unit for encoding data packets. It consists of k data packets and m check packets and is also the unit for data alignment. If a data commit operation occurs within the packet group boundary, it needs to be padded with zeros to the entire packet group.

[0033] A chunk group is a set of replicas obtained from the cluster master node. It is a container of a packet group and is also called an erasure coding chunk.

[0034] A data process / node is a node in a distributed storage system where data blocks are stored. It is typically responsible for operations such as writing, storing, reading, and deleting data block replicas.

[0035] Metadata nodes are nodes in a distributed storage system that store file metadata. They are typically used to store file status information and chunk location information, etc.

[0036] According to an embodiment of this application, a data processing method based on erasure coding is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The method embodiments provided in this application can be executed in a mobile terminal, computer terminal, or similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method based on erasure coding. As shown in Figure 1, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 1 is only illustrative and does not limit the structure of the above-described electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a different configuration than shown in Figure 1.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the erasure coding-based data processing method in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the erasure coding-based data processing method in the above embodiments. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display can be a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] According to an embodiment of this application, a data processing method based on erasure coding is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] Under the above operating environment, this application provides a data processing method based on erasure coding, as shown in Figure 2. Figure 2 is a flowchart of the data processing method based on erasure coding according to Embodiment 1 of this application. As shown in Figure 2, the method may include the following steps:

[0044] Step S201: If the client detects that there is target data to be sent, the target data is divided into multiple data packets according to the preset data packet size.

[0045] Optionally, the target data mentioned above refers to the data on the client side to be sent to data nodes in the distributed storage system. The target data can be data written by the user on the client side, or data that needs to be synchronously sent to other data nodes after a data node has been updated; no specific limitation is made in this application. Furthermore, the target data can be in the form of files, database records, streaming media data, etc. In a distributed storage environment, the target data can be generated or uploaded by the client and contains all the data information that the user needs to save. For example, when a user uploads a photo to a cloud storage service, the data of this photo is the aforementioned target data.

[0046] Optionally, the preset data packet size mentioned above is a strategy used in distributed storage systems to segment and process data. A data packet is the basic unit in erasure coding technology, representing smaller parts of data that have been segmented. The preset data packet size refers to the fixed-size blocks into which data is segmented during processing. This size is predetermined based on the specific needs and optimization goals of the system. For example, to improve parallel processing efficiency, the data packet size can be set to 1MB, 2MB, or larger. For instance, if the target data size is 5.5MB, segmenting it into 1MB data packets results in 6 data packets.

[0047] Step S202: If, among multiple data packets, the data volume in a target data packet is less than the preset data packet size, check data is generated based on the target data packet and erasure coding algorithm, and the padding data record of the target data packet is determined. The padding data record records the position offset information of the padding data required in the target data packet.

[0048] The data volume in the target data packet mentioned above is less than the preset data packet size, meaning the data volume in the target data packet does not meet the alignment requirement with the preset data packet size. This can also be understood as a data packet with a data volume less than that of a complete data packet group. The aforementioned position offset information may include position offset and length information. In this application, no padding data is actually written to the target data packet to meet the preset data packet size; that is, no padding data is actually written to the target data packet to achieve the alignment requirement. Instead, an erasure coding algorithm is used to calculate check data, and the position offset and length information of the padding data are recorded. For example, if the preset data packet size is 1MB, the target data size to be sent is 3MB + 128KB, and the target data is divided into 4 data packets, with 3 packets containing 1MB of data and the last packet containing only 128KB, check data will be generated based on the data of the 4 data packets, and the position offset and length information of the padding data will be recorded.<offset:128KB,length:896KB> The data records to be filled.

[0049] Optionally, generating verification data based on the target data packet and erasure coding algorithm includes: padding the target data packet with padding data; and generating verification data using the target data and padding data according to the erasure coding algorithm.

[0050] Optionally, the padding data mentioned above can be any data. The main purpose of the padding data is to ensure that the data packet meets the algorithm's requirement for a preset data packet size during erasure coding calculation. In practice, the padding data is not actually stored in the distributed storage system, but is "simulated" during the calculation of the checksum. For example, if the actual size of the target data packet is 0.5MB, while the preset data packet size is 1MB, the distributed storage system will use virtual padding data to expand the size of the target data packet to 1MB.

[0051] After the target data packet is padded to the preset data packet size, erasure coding algorithms can be used to calculate the complete data packet, which includes the actual data and the padding data, to generate checksum data. The erasure coding algorithm is a data redundancy technique that calculates checksum information for the original data, allowing the original data to be recovered from the remaining data and checksum information even if some data is lost. Although the padding data does not exist in physical storage, it is treated as present when calculating the checksum data.

[0052] For example, an erasure coding algorithm, EC(4,2), means that a distributed storage system requires four data packets and two parity data packets to form an erasure coding group. If the target data is divided into three 1MB data packets and one 0.5MB data packet, then the last 0.5MB data packet will be padded to 1MB using virtual padding data, and then used together with the previous three data packets to calculate the parity data using the erasure coding algorithm. In this application, by padding the data packets with padding data, and then using the target data and padding data to calculate the parity data, the integrity and recoverability of the data can still be maintained, thus improving the fault tolerance of the distributed storage system.

[0053] Besides the methods for generating checksum data mentioned above, other methods can also be used, mainly depending on the type and implementation details of the erasure coding. For example, Reed-Solomon codes (RS codes) can be used, a common type of erasure coding based on polynomial functions and finite field arithmetic. In RS codes, checksum data is generated by mapping the target data block to a specific polynomial and then calculating the remainder of the polynomial. The choice of how to generate checksum data depends on the specific requirements of the storage system, such as data recovery speed, storage efficiency, and computational complexity; therefore, no specific limitations are made in this application.

[0054] Step S203: Send the target data, verification data, and fill data records to multiple data nodes for storage.

[0055] Through the above steps, the target data, verification data, and padding data records are sent to multiple data nodes in the distributed storage system for storage. This avoids sending the target data packet to the data nodes after padding it with data, thereby reducing the storage space occupied and achieving the technical effect of improving storage efficiency and minimizing the system's storage overhead.

[0056] Optionally, Figure 3 is a schematic diagram of an optional erasure coding-based data processing method. The client divides the user data (corresponding to the target data mentioned above) into each replica according to the packet size. For portions not aligned with the packet group (i.e., portions not meeting the preset packet size), padding data is used to fill the gaps, and a checksum data block is generated. When the client sends data, it does not send the padding data to the data node. Instead, it records the padding data's position offset information (position offset and length information) and sends this padding record along with the user data to the data node. The data node stores all data in the data block and the padding data record. When a node fails and the client needs to read the padding data to perform data recovery, the data node finds that the current data offset is padding data and no actual data is stored. It then generates the corresponding data and returns it to the data recovery process.

[0057] In the above technical solution, there is no need to store fill data in the data nodes; only data fill records need to be stored on the nodes. For example...<offset:128KB,length:896KB> This reduces the waste of storage space caused by data filling records.

[0058] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0060] Example 2

[0061] Figure 4 is a flowchart of a data processing method based on erasure coding according to Embodiment 2 of this application. As shown in Figure 4, the method may include the following steps:

[0062] Step S401: Receive the target data, verification data, and padding data records sent above;

[0063] After the data nodes in the server receive the target data, verification data, and padding data records sent by the client, each data node can be responsible for storing a data packet or verification data, while the padding data records will be stored on the metadata node or the node stored with the data packet for later use.

[0064] Optionally, the following storage method provided in Embodiment 2 of this application can ensure data integrity and recoverability while minimizing storage space waste. For example, the storage steps can be as follows:

[0065] 1. Data storage partitioning: Data nodes first store the received target data according to the corresponding data packets. Each data packet is assigned to a specific storage partition or location to ensure efficient data access and management.

[0066] 2. Padding Data Record Storage: For misaligned data portions of data packets, data nodes do not directly store the actual padding data. Instead, they store padding data records in the metadata area. Padding data records typically include the offset and length information of the padding data, used as a reference for subsequent data recovery.

[0067] 3. Erasure coding data update: After storing the target data, the data node updates the corresponding check data according to the erasure coding algorithm. If it is an unaligned write, only the stored target data portion is used for check data calculation, while the information in the fill data record is not included in the calculation.

[0068] 4. Metadata Maintenance: Data nodes need to maintain metadata associated with the stored data, including populate data records. Metadata records information such as the stored data packets, the location offset and length of the populate data, and the mapping relationship of the checksum data, which helps to quickly locate and recover data.

[0069] 5. Data Consistency Check: After storing data, data nodes can perform a data consistency check to ensure that the stored data blocks (including target data and check data) conform to the erasure coding rules, and at the same time check the integrity and accuracy of the populated data records.

[0070] 6. Data Recovery Preparation: During the data recovery or re-encoding phase, the data nodes will automatically generate the required padding data based on the position offset and length information in the padding data records to supplement the insufficient parts of user data during non-aligned writing, ensuring the integrity and accuracy of data recovery.

[0071] Through the storage steps described above, data nodes can efficiently store target data sent by clients. Simultaneously, by storing filler data records instead of actual filler data, redundant space usage is reduced, achieving effective storage space management and cost optimization. During data recovery, the position offset and length information in the filler data records help quickly generate filler data, restoring the integrity of the original data, without actually storing this filler data, thus saving storage resources.

[0072] Step S402: Based on the position offset information in the fill data record, generate the fill data required for the target data packet to complete the target data packet;

[0073] In this application, the aforementioned position offset information may include position offset and length information. Position offset refers to the starting position of the padding data, relative to the start point of the data packet. For example, if the preset data packet size corresponding to the target data packet is 1MB (i.e., the alignment size requirement is 1M), but the target data packet size is only 512KB, then the position offset will indicate that padding data needs to start from the 512KB position. The aforementioned length information indicates the amount of data to be padded, i.e., how many bytes need to be occupied by virtual padding data. The length information will indicate that an additional 512KB of data needs to be padded. The padding data required to generate the target data packet is 512KB of data, and the generated 512KB of data is used to complete the target data packet.

[0074] Step S403: Perform the target operation based on the target data, the fill data, and the verification data.

[0075] After generating the required padding data, target operations can be performed based on the target data, padding data, and checksum data, such as data recovery, re-encoding, or data reading. For example, during data recovery, dynamically generating padding data ensures the integrity of the data packet group, thereby accurately recovering lost data and maintaining data integrity and consistency.

[0076] In an optional embodiment, as shown in Figure 5, the position offset information includes: position offset and length information. Performing the target operation based on the target data, padding data, and checksum data includes:

[0077] Step S501: If the target operation is data recovery, the lost data portion is recovered using the target data and check data through erasure coding algorithm to generate a complete data packet.

[0078] Optionally, if some data packets are lost, a data recovery operation may be triggered. The distributed storage system can use erasure coding algorithms to recover the lost data packets based on the remaining unlost data packets and checksum data.

[0079] For example, an erasure coding set contains four data packets and two checksums. The third data packet (1MB in size) is lost, while the actual size of the four data packets is 3.5MB. During data recovery, the distributed storage system reads the padding data records and finds 0.5MB of virtual padding data in the fourth data packet. To ensure the accuracy of the recovered data, the system ignores this virtual padding data and does not use it to recover the lost data. Instead, it uses only the two complete data packets, the 0.5MB of actual data from the fourth data packet, and the two checksums to generate a complete set of four data packets, thus recovering the lost third data packet.

[0080] In step S502, if the target operation is re-encoding, the target data and padding data are treated as a complete data packet. Based on the erasure coding algorithm, the position offset and length information of the target data and padding data, and the check data are used to recalculate new check data to update the check information.

[0081] Optionally, when the distributed storage system needs to change the erasure coding configuration, a recoding operation may be triggered. For example, increasing or decreasing the number of parity data blocks. In this case, the distributed storage system will treat the padding data as part of the actual data to ensure data integrity, and calculate the new parity data based on the position offset and length information in the padding data record.

[0082] For example, a distributed storage system might need to change the erasure coding configuration from EC(4,2) to EC(5,2) in the example above, essentially adding a data packet. Before recoding, the last data packet (the 4th data packet) has 0.5MB of virtual padding data. During recoding, the distributed storage system can treat the 0.5MB of virtual padding data from the 4th data packet as part of the complete data packet, generate new checksum data according to the new erasure coding configuration, and simultaneously create a new data packet to store the additional data or virtual padding data, ensuring data integrity and the continuity of the erasure coding.

[0083] Step S503: If the target operation is data reading, combine the target data with the padding data to generate a complete data packet to meet the data format requirements of the read request.

[0084] Optionally, a data read operation can be triggered when stored data needs to be returned to the client or user. Virtual padding data can be generated based on information in the padding data record to fill in misaligned data packets, ensuring that the packet size conforms to a preset format for easier subsequent processing by the client or user.

[0085] For example, a client initiates a data read request, requesting to read 3.5MB of stored data. During the data read, the distributed storage system generates 0.5MB of virtual padding data in the fourth data packet based on the padding data records. This virtual padding data is then used to fill the 0.5MB data packet. The distributed storage system can then send all four data packets (three 1MB actual data packets and one 1MB data packet, where 0.5MB is user data and 0.5MB is virtual padding data) to the client together to meet the integrity and format requirements of the data read.

[0086] Optionally, during the data recovery or re-encoding stage, after the data node generates the padding data required for the target data packet based on the position offset and length information in the padding data record, the next step may be to perform the following steps:

[0087] 1. Data Reassembly: Data nodes use user data, generated padding data, and stored checksum data to reassemble the entire data packet group (or data block). This is achieved through reverse computation using erasure coding algorithms, ensuring that even if some data is lost, the complete data group can be recovered from the remaining data and checksum information.

[0088] 2. Data Validation: After data reassembly, data nodes will validate the reassembly to ensure its integrity and accuracy. This may include checking the data checksum or reapplying erasure coding algorithms to confirm that the data matches the expected checksum.

[0089] 3. Data Rewriting: If a data node detects an anomaly, such as incomplete data or failed verification, it re-encodes the data, generates new verification data, and rewrites this data to the affected data node or other secure data nodes. This process can be automatic, triggered by the distributed storage system when it detects data loss or node failure.

[0090] 4. Status Update: After data recovery or re-encoding is complete, the data node updates its metadata status to reflect the latest data distribution. This may involve updating and populating data records, verifying data location information, and any other metadata related to data recovery.

[0091] 5. Notification and Synchronization: After data nodes complete data recovery, they will notify other relevant nodes and metadata nodes in the distributed storage system to update their data status, ensuring data consistency throughout the distributed storage system. Simultaneously, it may be necessary to synchronize data with other nodes to update or supplement their storage.

[0092] By employing the information in the padding data records, lost data can be accurately recovered from remaining data packets and checksum data, improving the speed and efficiency of data recovery while ensuring data integrity. The padding data records also allow for flexible re-encoding of existing data without actually storing the padding data when changing erasure coding configurations, adapting to different storage needs and redundancy strategies. Furthermore, dynamically generating virtual padding data ensures the format requirements and data integrity during data reading, enabling clients or users to correctly receive and parse data. In summary, by using the position offset and length information in the padding data records to guide data recovery, re-encoding, and data reading operations, the accuracy of data processing is guaranteed.

[0093] In an optional embodiment, if the padding data record includes a target padding method, generating the padding data required to complete the target data packet based on the position offset information in the padding data record includes:

[0094] Based on the position offset, length information, and target padding method in the padding data record, the padding data required for the target data packet is generated to complete the target data packet.

[0095] Optionally, the aforementioned padding data record may include a target padding method, which provides more specific and flexible guidance for the padding data required to generate the target data packet. The target padding method can specify the type or pattern of padding data, such as all-zero byte padding, padding with a specific pattern, or random data padding.

[0096] The operation steps can be as follows: Read the padding data record: Before performing data recovery, re-encoding, or data reading operations, the data process / node reads the padding data record to obtain the position offset, length information, and target padding method. Generate padding data based on the padding data record: Based on the target padding method provided in the padding data record, the data node generates padding data with a specific pattern or type at the corresponding position to fill the target data packet to the preset alignment size. For example, if the target padding method is all-zero bytes, then a specified length of all-zero bytes will be added after the starting point indicated by the position offset.

[0097] For example, consider four data packets (each 1MB). The fourth packet actually contains 0.5MB of user data. The padding data record includes not only the position offset (starting at 0.5MB) and length information (0.5MB), but also specifies the target padding method as all zero bytes. When the distributed storage system needs to perform data recovery operations: the data process / node reads the padding data record to obtain all necessary information, including the padding method of all zero bytes. When recovering lost data packets, the data node uses the remaining three data packets and checksum data according to the erasure coding algorithm to recover the data. For the fourth data packet, it will generate 0.5MB of all-zero-byte padding data, which will be combined with the existing 0.5MB of user data to form a complete 1MB data packet.

[0098] The above technical solution, by specifying the target filling method, can more accurately construct data packets when data recovery is required, ensuring that the recovered data packets have the same format and structure as the original data packets, thereby improving the accuracy of data recovery and data consistency.

[0099] In one alternative embodiment, the padding data record can be stored on the data node in the form of metadata, which includes at least: the position offset of the padding data, length information, and the identifier of the target data packet.

[0100] Optionally, the aforementioned metadata includes, but is not limited to, the location offset, length information, and associated packet identifier of the padding data, which can ensure that when performing the target operation, the data node can quickly and accurately locate and generate the required virtual padding data.

[0101] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0103] Example 3

[0104] According to Embodiment 3 of this application, an apparatus for implementing the erasure coding-based data processing method in Embodiment 1 is also provided, as shown in FIG6. The apparatus includes: a segmentation unit 601, a determination unit 602, and a sending unit 603.

[0105] Specifically, the segmentation unit 601 is configured to segment the target data according to a preset data packet size to obtain multiple data packets when the client detects that there is target data to be sent.

[0106] The determining unit 602 is configured to, in multiple data packets, if the data volume in a target data packet is less than the preset data packet size, calculate and generate verification data based on the target data packet and erasure coding algorithm, and determine the padding data record of the target data packet, wherein the padding data record records the position offset information of the padding data required in the target data packet;

[0107] The sending unit 603 is configured to send target data, verification data and fill data records to multiple data nodes for storage.

[0108] In the erasure coding-based data processing apparatus provided in Embodiment 3 of this application, the segmentation unit 601, upon detecting the presence of target data to be sent on the client, segments the target data according to a preset data packet size to obtain multiple data packets. The determination unit 602, among the multiple data packets, if the data volume of any target data packet is less than the preset data packet size, calculates and generates verification data based on the target data packet and the erasure coding algorithm, and determines the padding data record for the target data packet. The padding data record records the position offset information of the required padding data in the target data packet. The sending unit 603 sends the target data, verification data, and padding data record to multiple data nodes for storage. This solves the technical problem that, during data writing and recovery using erasure coding technology, padding data needs to be inserted when processing non-aligned data, leading to increased redundant storage space usage. In this application, sending the target data, verification data, and padding data record to multiple data nodes for storage avoids sending the target data packet after padding, thereby reducing storage space usage, improving storage efficiency, and minimizing system storage overhead.

[0109] Optionally, in the erasure coding-based data processing apparatus provided in Embodiment 3 of this application, the determining unit 602 includes: a filling module, configured to fill the target data packet with filling data; and a calculation module, configured to calculate and generate verification data using the target data and the filling data according to the erasure coding algorithm.

[0110] It should be noted that the segmentation unit 601, determination unit 602, and sending unit 603 mentioned above correspond to steps S201 to S203 in the above embodiments. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in Embodiment 1.

[0111] It should be noted that the preferred implementation schemes involved in the above embodiment one of this application are the same as the schemes, application scenarios and implementation processes provided in the above embodiment one, but are not limited to the schemes provided in the above embodiment one.

[0112] Example 4

[0113] According to an embodiment of this application, an apparatus for implementing the erasure coding-based data processing method in Embodiment 2 is also provided, as shown in FIG7. The apparatus includes a receiving unit 701, a generating unit 702, and an execution unit 703.

[0114] Specifically, the receiving unit 701 is configured to receive the target data, verification data, and padding data records sent above;

[0115] The generation unit 702 is configured to generate padding data required to complete the target data packet;

[0116] Execution unit 703 is configured to perform target operations based on target data, padding data, and verification data.

[0117] In the erasure coding-based data processing apparatus provided in Embodiment 4 of this application, the receiving unit 701 receives the aforementioned target data, check data, and padding data records; the generating unit 702 generates the padding data required for the target data packet to complete the target data packet; and the execution unit 703 executes the target operation based on the target data, padding data, and check data. This solves the technical problem that in the data writing and recovery process using erasure coding technology, padding data needs to be inserted when processing unaligned data, leading to increased redundant storage space usage. In this application, the target data, check data, and padding data records sent by the client are received and stored, avoiding the need to pad the target data packet before sending it to the data node for storage, thereby reducing storage space usage and achieving the technical effect of improving storage efficiency and minimizing system storage overhead.

[0118] Optionally, in the erasure coding-based data processing apparatus provided in Embodiment 4 of this application, the position offset information includes: position offset and length information. The execution unit 703 includes: a recovery module, configured to, if the target operation is data recovery, recover the lost data portion using the target data and check data through the erasure coding algorithm to generate a complete data packet; an encoding module, configured to, if the target operation is re-encoding, treat the target data and padding data as a complete data packet, and, according to the erasure coding algorithm, recalculate new check data using the position offset and length information of the target data and padding data, and the check data to update the check information; and a reading module, configured to, if the target operation is data reading, combine the target data and padding data to generate a complete data packet to meet the data format requirements of the reading request.

[0119] Optionally, in the erasure coding-based data processing apparatus provided in Embodiment 4 of this application, the generation unit 702 includes: a padding module, configured to generate padding data required for the target data packet to pad the target data packet if the padding data record includes a target padding method, based on the position offset, length information and target padding method in the padding data record.

[0120] Optionally, in the erasure coding-based data processing apparatus provided in Embodiment 4 of this application, the padding data record is stored on the data node in the form of metadata, which includes at least: the position offset of the padding data, length information, and the identifier of the target data packet.

[0121] It should be noted that the receiving unit 701, generating unit 702, and execution unit 703 mentioned above correspond to steps S401 to S403 in the above embodiments. The three units and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in Embodiment 1.

[0122] It should be noted that the preferred implementation scheme involved in the above embodiment four of this application is the same as the scheme, application scenario and implementation process provided in the above embodiment two, but is not limited to the scheme provided in the above embodiment two.

[0123] Example 5

[0124] Embodiment 5 of this application can provide an electronic device, which can be any electronic device in a group of electronic device terminals. Optionally, in this embodiment, the above-mentioned electronic device can also be replaced by a terminal device such as a mobile terminal.

[0125] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0126] In this embodiment, the electronic device described above can execute the program code of the following steps in the erasure coding-based data processing method provided in Embodiment 5: when the client detects that there is target data to be sent, the target data is divided into multiple data packets according to a preset data packet size; if the amount of data in the target data packet is less than the preset data packet size, check data is calculated and generated according to the target data packet and the erasure coding algorithm, and the padding data record of the target data packet is determined, wherein the padding data record records the position offset information of the padding data required in the target data packet; the target data, check data and padding data record are sent to multiple data nodes for storage.

[0127] The aforementioned electronic device can execute the program code for the following steps in the erasure coding-based data processing method provided in Embodiment 5: calculating and generating verification data based on the target data packet and the erasure coding algorithm includes: padding the target data packet with padding data; and calculating and generating verification data using the target data and padding data according to the erasure coding algorithm.

[0128] The aforementioned electronic device can execute the program code of the following steps in the erasure coding-based data processing method provided in Embodiment 5: receiving the target data, check data, and padding data record sent above; generating padding data required for the target data packet to complete the target data packet based on the position offset information in the padding data record; and performing the target operation based on the target data, padding data, and check data.

[0129] The aforementioned electronic device can execute the program code for the following steps in the erasure coding-based data processing method provided in Embodiment 5: performing a target operation based on target data, padding data, and checksum data includes: if the target operation is data recovery, recovering the lost data portion using the target data and checksum data through the erasure coding algorithm to generate a complete data packet; if the target operation is re-encoding, treating the target data and padding data as a complete data packet, and recalculating new checksum data using the position offset and length information of the target data and padding data, and checksum data, according to the erasure coding algorithm, to update the checksum information; if the target operation is data reading, combining the target data and padding data to generate a complete data packet to meet the data format requirements of the reading request.

[0130] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on erasure coding provided in Embodiment 5: If the padding data record includes a target padding method, generate the padding data required for the target data packet to complete the target data packet according to the position offset information in the padding data record. This includes: generating the padding data required for the target data packet to complete the target data packet according to the position offset, length information and target padding method in the padding data record.

[0131] The aforementioned electronic device can execute the program code of the following steps in the erasure coding-based data processing method provided in Embodiment 5: the padding data record is stored on the data node in the form of metadata, and the metadata includes at least: the position offset of the padding data, length information and the identifier of the target data packet.

[0132] Optionally, FIG8 is a structural block diagram of an electronic device according to Embodiment 5 of this application. As shown in FIG8, the electronic device 800 may include: one or more (only one is shown in FIG8) processors 802 and memory 804. The electronic device 800 may also include a storage controller for controlling and managing the memory 804; the electronic device 800 may also include a peripheral interface for connecting to a radio frequency module, an audio module, and a display screen, etc.

[0133] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the erasure coding-based data processing method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned erasure coding-based data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the electronic device 800 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0134] The processor can access information and applications stored in memory via the transmission device to perform the following steps: Upon detecting that the client has target data to be sent, the target data is segmented according to a preset data packet size to obtain multiple data packets; if the data volume in any of the target data packets is less than the preset data packet size, check data is calculated and generated based on the target data packet and the erasure coding algorithm, and a padding data record for the target data packet is determined, wherein the padding data record records the position offset information of the required padding data in the target data packet; the target data, check data, and padding data record are sent to multiple data nodes for storage.

[0135] Optionally, the processor may also execute program code that performs the following steps: calculating and generating verification data based on the target data packet and erasure coding algorithm, including: padding the target data packet with padding data; and calculating and generating verification data using the target data and padding data according to the erasure coding algorithm.

[0136] Optionally, the processor may also execute program code that performs the following steps: receiving the target data, check data, and padding data records sent above; generating padding data required for the target data packet to complete the target data packet based on the position offset information in the padding data records; and performing the target operation based on the target data, padding data, and check data.

[0137] Optionally, the processor may also execute program code with the following steps: performing a target operation based on target data, padding data, and checksum data, including: if the target operation is data recovery, recovering the lost data portion using the target data and checksum data through an erasure coding algorithm to generate a complete data packet; if the target operation is re-encoding, treating the target data and padding data as a complete data packet, and recalculating new checksum data using the position offset and length information of the target data and padding data, and the checksum data, according to the erasure coding algorithm, to update the checksum information; if the target operation is data reading, combining the target data and padding data to generate a complete data packet to meet the data format requirements of the read request.

[0138] Optionally, the processor may also execute program code that performs the following steps: if the padding data record includes a target padding method, generate padding data required for the target data packet to complete the target data packet based on the position offset information in the padding data record, including: generating padding data required for the target data packet to complete the target data packet based on the position offset, length information and target padding method in the padding data record.

[0139] Optionally, the processor may also execute program code that performs the following steps: further, the padding data record is stored on the data node in the form of metadata, which includes at least: the position offset of the padding data, length information, and the identifier of the target data packet.

[0140] It will be understood by those skilled in the art that the structure shown in FIG8 is merely illustrative, and the electronic device 800 may also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. FIG8 does not limit the structure of the aforementioned electronic device. For example, the electronic device 800 may also include more or fewer components (such as network interfaces, display devices, etc.) than shown in FIG8, or have a different configuration than that shown in FIG8.

[0141] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0142] Example 6

[0143] Embodiment Six of this application also provides a computer-readable storage medium. Optionally, in this embodiment, the computer program product described above can be used to store the program code executed by the erasure coding-based data processing method provided in Embodiment One.

[0144] Example 7

[0145] Embodiment 7 of this application also provides a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the erasure coding-based data processing method provided in Embodiment 1.

[0146] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0147] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0148] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Specifically, the fifth embodiment of the apparatus described above is merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0153] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method based on erasure coding, comprising: If the client detects that there is target data to be sent, the target data is divided into multiple data packets according to a preset data packet size; If, among the plurality of data packets, the amount of data in a target data packet is less than the preset data packet size, verification data is generated based on the target data packet and the erasure coding algorithm, and a padding data record for the target data packet is determined, wherein the padding data record records the position offset information of the padding data required in the target data packet; The target data, the verification data, and the fill data record are sent to multiple data nodes for storage.

2. The method according to claim 1, wherein, The verification data generated based on the target data packet and erasure coding algorithm includes: The target data packet is padded with padding data; According to the erasure coding algorithm, the target data and the padding data are used to calculate and generate verification data.

3. The method according to claim 2, wherein, The verification data generated based on the target data packet and erasure coding algorithm includes: The target data packet is mapped onto a polynomial, and the remainder of the polynomial is calculated to generate the verification data.

4. The method according to claim 1, wherein, The position offset information includes: position offset and length information.

5. The method according to claim 2, wherein, Filling the target data packet with padding data includes: The size of the target data packet is expanded to the preset data packet size by virtually padding data.

6. A data processing method based on erasure coding, comprising: Receive the target data, verification data, and fill data record sent in claim 1 or claim 2; Based on the position offset information in the fill data record, generate the fill data required for the target data packet to complete the target data packet; The target operation is performed based on the target data, the fill data, and the verification data.

7. The method according to claim 6, wherein, The position offset information includes: position offset and length information. Performing the target operation based on the target data, the padding data, and the verification data includes: If the target operation is data recovery, the lost data portion is recovered using the target data and the verification data through erasure coding algorithm, and a complete data packet is generated. If the target operation is re-encoding, the target data and the padding data are regarded as a complete data packet. Based on the erasure coding algorithm, the position offset and length information of the target data and the padding data, and the check data are used to recalculate new check data to update the check information. If the target operation is data reading, the target data is combined with the padding data to generate a complete data packet to meet the data format requirements of the read request.

8. The method according to claim 7, wherein, If the padding data record includes a target padding method, generating the padding data required to complete the target data packet based on the position offset information in the padding data record includes: Based on the position offset, length information, and target padding method in the padding data record, the padding data required to complete the target data packet is generated.

9. The method according to claim 7, wherein, The padding data record is stored on the data node in the form of metadata, which includes at least: the position offset of the padding data, length information, and the identifier of the target data packet.

10. The method according to claim 7, wherein, Receiving the target data, verification data, and padding data record sent in claim 1 or claim 2 includes: The server receives the target data, the verification data, and the fill data record through its data nodes. The target data is stored through the data node, and the populated data record is stored in the metadata area.

11. The method according to claim 8, wherein, The target filling method is to specify the type or pattern of the filling data.

12. The method according to claim 10, wherein, The method further includes: Before performing data recovery, re-encoding, or data reading operations, the data node is used to read the filling data record to obtain the position offset, length information, and target filling method. Based on the target padding method provided in the padding data record, padding data is generated at the corresponding position to supplement the target data packet to the preset alignment size.

13. The method according to claim 6, wherein, After generating the padding data required for the target data packet based on the position offset and length information in the padding data record, the method further includes: The data packet is reassembled based on the target data packet, the generated padding data, and the stored checksum data.

14. A data processing apparatus based on erasure coding, comprising: The segmentation unit is configured to, upon detecting that the client has target data to be sent, segment the target data according to a preset data packet size to obtain multiple data packets; The determining unit is configured to, among the plurality of data packets, if there is a target data packet whose data volume is less than a preset data packet size, calculate and generate verification data based on the target data packet and the erasure coding algorithm, and determine the padding data record of the target data packet, wherein the padding data record records the position offset information of the padding data to be filled in the target data packet; The sending unit is configured to send the target data, the verification data, and the filling data record to multiple data nodes for storage.

15. An electronic device comprising: Memory, which stores executable programs; The processor, connected to the memory via a bus, is used to run the program, wherein the program executes the following steps during runtime: when target data to be sent is detected on the client, the target data is divided into multiple data packets according to a preset data packet size; If, among the plurality of data packets, the amount of data in a target data packet is less than the preset data packet size, verification data is generated based on the target data packet and the erasure coding algorithm, and a padding data record for the target data packet is determined, wherein the padding data record records the position offset information of the padding data required in the target data packet; The target data, the verification data, and the fill data record are sent to multiple data nodes for storage.

16. The electronic device according to claim 15, wherein, The program also performs the following steps when it runs: The target data packet is padded with padding data; According to the erasure coding algorithm, the target data and the padding data are used to calculate and generate verification data.

17. The electronic device according to claim 15, wherein, The program also performs the following steps when it runs: The target data packet is mapped onto a polynomial, and the remainder of the polynomial is calculated to generate the verification data.

18. The electronic device according to claim 15, wherein, The program also performs the following steps when it runs: Receive target data, verification data, and populate data records; Based on the position offset information in the fill data record, generate the fill data required for the target data packet to complete the target data packet; The target operation is performed based on the target data, the fill data, and the verification data.

19. The electronic device according to claim 18, wherein, The program also performs the following steps when it runs: The position offset information includes: position offset and length information. Performing the target operation based on the target data, the padding data, and the verification data includes: If the target operation is data recovery, the lost data portion is recovered using the target data and the verification data through erasure coding algorithm, and a complete data packet is generated. If the target operation is re-encoding, the target data and the padding data are regarded as a complete data packet. Based on the erasure coding algorithm, the position offset and length information of the target data and the padding data, and the check data are used to recalculate new check data to update the check information. If the target operation is data reading, the target data is combined with the padding data to generate a complete data packet to meet the data format requirements of the read request.

20. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the erasure coding-based data processing method according to any one of claims 1 to 13.

Images