Data synchronization method and related product

Abstract

A data synchronization method and a related product, which are applied to the technical field of storage. In the present application, both a first device and a second device store data of a file, the first device can manage reading and writing of the file on a plurality of devices at the granularity of data blocks on the basis of extended metadata of the file, and the extended metadata comprises last modification times corresponding to a plurality of data blocks of the file. When the file on the second device is modified, the second device indicates a data location of the modification and an operation time of the modification to the first device. The first device determines, on the basis of the data location of the modification, a target data block involved in the modification, and compares the last modification time of the target data block with the operation time of the modification. If the operation time is later than the last modification time, the first device acquires updated data from the second device and updates the file on the first device. In the present application, modifications to a file by a device are merged at the granularity of data blocks, and collaborative update of file data by multiple devices is supported, thereby solving the problem of additional occupation of storage space during conflicting modifications.

Description

A data synchronization method and related products

[0001] This application claims priority to Chinese Patent Application No. 202510052999.2, filed on January 10, 2025, entitled "A Data Synchronization Method and Related Products", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of storage technology, and in particular to a data synchronization method and related products. Background Technology

[0003] With the development of network technology and the popularization of cloud architecture, single storage devices can no longer meet the needs of some businesses, and more and more users are storing data on multiple storage devices. For storage systems that include multiple storage devices, such as global file systems, some files may be stored on multiple storage devices, and multiple storage devices may modify the file system data, making the synchronization of file system data across multiple storage devices a challenge. Especially when multiple storage devices modify the same file, file modification conflicts will occur in the system, and resolving file modification conflicts is a critical step in the data synchronization process.

[0004] Currently, the method to resolve file modification conflicts is to split the conflicting file into two separate files. However, splitting it into two files doubles the storage space, increasing costs. Moreover, from the user's perspective, the original single file has been split into multiple files, usually requiring manual deletion of unwanted versions and renaming, resulting in a poor user experience. Summary of the Invention

[0005] This application provides a data synchronization method and related products that can merge device modifications to files at the data block level, support collaborative updates of file data in the file system by multiple devices, realize file synchronization between multiple storage systems, solve the problem of extra storage space occupation caused by conflicting file modifications, and improve the user experience.

[0006] In one aspect, this application provides a data synchronization method applied to a first device, for example executed by the first device or a module within the first device. For ease of understanding, the following description uses the first device as the executing entity. The first device manages the reading and writing of the first file across multiple devices at the data block granularity based on first extended metadata of the first file. The first extended metadata includes the last modification time corresponding to each of the multiple data blocks of the first file. The multiple devices include the first device and a second device.

[0007] Furthermore, the first device stores the data of the first file, and some or all of the data of the first file is also stored on the second device. For example, on the second device, some or all of the data of the first file can be stored in a cached form or in a persistent storage form. In summary, the data of the first file is stored on both the first and second devices, and the data of the first file needs to remain consistent between the first and second devices.

[0008] The data synchronization method includes: a first device acquiring a first update message from a second device, the first update message indicating that a first file stored on the second device has undergone a first modification. The first update message includes a first modification location and a first modification time; the first modification location indicates the position of the first data corresponding to the first modification in the first file, and the first modification time indicates the operation time of the first modification. The first device determines a target data block among multiple data blocks, the target data block being the data block corresponding to the first data.

[0009] Furthermore, if the first device obtains the first data from the second device when the first modification time is later than the last modification time corresponding to the target data block, and uses the first data to update the first file stored on the first device.

[0010] The file's data is managed in the form of multiple data blocks, each indicating a range of locations. When a file is modified, the newly modified data corresponds to the modified data. The modification location indicates the position of the modified data within the first file, thus determining which data blocks were involved in the first modification. The modification time indicates the exact moment or time elapsed, and can be either a timestamp or a version number.

[0011] In the above embodiment, the file data is stored on both the first and second devices. The file data is managed in the form of data blocks, and the last modification time corresponding to each data block is recorded on the first device. When the first file stored on the second device is modified, the second device sends an update message to the first device. The update message indicates the location of the modification and the time of the modification operation. Based on the location of the modification, the first device finds the last modification time of the target data block corresponding to that location and compares the last modification time with the time when the modification operation was performed by the second device. If the time when the modification operation was performed is later than the local last modification time of the target data block, it indicates that the first modification is a new modification. The first device then retrieves the first data from the second device and updates the first file stored on the first device, thus successfully modifying the data of the first file.

[0012] In this embodiment, the first device maintains the last modification time of multiple data blocks corresponding to the first file. When a modification occurs, the time is used as the basis for updating the data block content, enabling the merging of data modifications made to the first file by other devices at the data block level. Similarly, other devices can also send (or publish) update messages when modifying the first file, and the first device also retains the later modifications based on time comparison. Therefore, this application can support collaborative updates of file data in the file system by multiple devices, realizing file synchronization between multiple storage systems.

[0013] The embodiments of this application can also greatly improve the problem of data loss when file modification conflicts occur. Specifically, for file modification conflicts, due to different degrees of conflict, the following two possible situations may exist:

[0014] First, different data blocks of the same file are modified. For example, device S1 modifies data block B1 of the first file, while device S2 modifies data block B2 of the first file. Since time comparison and data updates are performed at the data block level, this situation does not conflict at the data block level. Therefore, the first device can update data block B1 with the modified data from device S1 and update data block B2 with the modified data from device S2, ensuring that modifications from multiple devices are all preserved. Therefore, this application manages file data at the data block level, avoiding the problem of modification loss when file modification conflicts occur, and improving the user experience.

[0015] Second, regarding modifications to the same data block of the same file, for example, device S1 modifies data block B1 of the first file at time T1, while device S2 modifies data block B1 of the first file at time T2. Since time comparison and data updates are performed at the data block granularity, after processing the above modifications, the content of the last retained data block B1 by the first device corresponds to the latest modification time among time T1, time T2, and the last local modification time. In other words, the data block of the first file stored in the first device contains the latest data content. In other words, even in the event of a data block-level modification conflict, this application will not generate a new file, thus avoiding increased costs.

[0016] In summary, this application can merge device modifications to files at the data block level, support collaborative updates of file data in the file system by multiple devices, realize file synchronization within multiple device systems, solve the problem of extra storage space occupation caused by conflicting file modifications, and improve the user experience.

[0017] It should be understood that the first file is managed in the form of multiple data blocks, which are observed at the management level. Therefore, the division of data blocks is logical and does not necessarily mean that the data is stored in the actual physical form of multiple blocks. In other words, the device does not need to physically divide the first file; it simply manages the first file using a finer granularity (i.e., data blocks). Therefore, these data blocks can be understood as virtual data blocks used for more granular management of the first file.

[0018] In one possible implementation of the first aspect, the first file is stored on a device, including two cases: Case 1, the device is a storage device, including storage units, and the first file is stored in the storage units of the device. Case 2, the device is a computing device, and the first file is stored on a storage device accessible to the device or a storage device managed by the device, the storage device including storage units.

[0019] For example, the aforementioned first device and second device may be storage devices. As another example, the aforementioned first device and second device may also be devices with only computing capabilities, in which case the first file may be stored on a storage device that can be accessed or managed by the first device and second device.

[0020] In another possible implementation of the first aspect, the first extended metadata of the first file is maintained or managed by the first device. For example, the first device is capable of acquiring and updating the first extended metadata of the first file. Optionally, the first extended metadata of the first file may be stored on the first device or on other devices accessible to the first device.

[0021] In some cases, update messages, such as the aforementioned first update message and other update messages below, may also include one or more of the following: the identifier of the operating device, the identifier of the first file on the operating device, the identifier of the device to which the first file belongs, or the identifier of the first file on the device to which it belongs. Here, the operating device refers to the device that performed the modification, also known as the modification source, modification operation site, etc., so that the first device can know which device modified the file. Furthermore, the first device can also obtain the modified, latest data from the device that performed the modification based on the identifier of the operating device. For example, if the device that performed the first modification is the second device, then the first update message may also include the identifier of the second device to indicate that the device that performed the first modification is the second device.

[0022] The identifier of the first file on the operating device is used to identify the first file on the operating device, so that other devices can retrieve the modified file data from the operating device. For example, it could be a universally unique identifier (UUID) for the first file on the operating device. For instance, when the first file stored on the second device is modified, the identifier of the first file on the operating device becomes the identifier of the first file on the second device. In some cases, each device maintains a local identifier for the first file, which facilitates the management of the first file's data content while maintaining compatibility with the respective namespaces of each device. Other devices can retrieve the data of the first file from the operating device using the identifier of the first file on the operating device.

[0023] The identifier of the device that owns the first document is used to identify the device that owns the first document, so that the device that owns the first document can manage the data of the first document, such as receiving messages related to the first document. In some cases, the device that owns the first document is used to process update messages related to the first document, and carrying the identifier of the device that owns the first document in the update message helps the device that owns the first document maintain the data content of the first document.

[0024] The identifier of the first file on the home device is used to identify the first file on the home device so that the home device can find the file's data and / or the file's extended metadata, such as the first file's UUID on the home device.

[0025] In one possible implementation of the first aspect, the data size of the first file is greater than 1MB. Each data block in at least one data block is larger than 1MB. By designing the data block granularity to be smaller, the first device can manage the file with finer granularity, thereby reducing the probability of data block conflicts.

[0026] In one possible implementation of the first aspect, the data size of the first file is further greater than 4MB, and the size of each data block in at least one data block is greater than 4MB. With a larger granularity of data blocks, the space occupied by the extended metadata of the first file is reduced, thus reducing the storage cost of the file.

[0027] In some cases, multiple data blocks occupy the same size. For example, each data block may be 4MB in size.

[0028] In some cases, the size of each data block in multiple data blocks is not equal. For example, the size of the data blocks in a file is predefined, such as cycling between 4M and 2M.

[0029] In another possible implementation of the first aspect, the method further includes: the first device updating the last modification time corresponding to the target data block in the first extended metadata to a first modification time. In the above implementation, the first device updates the last modification time of the target data block to the latest first modification time, maintaining the accuracy and real-time performance of the extended metadata of the first file and improving system stability.

[0030] In another possible implementation of the first aspect, the method further includes: a first device sending a first modification confirmation message, the first modification confirmation message being used to instruct the first device to confirm the first modification.

[0031] In the above embodiments, the first device can issue a confirmation of modification to the first file, which on the one hand enables the second device to know whether the modification operation has been successfully accepted, and on the other hand enables other devices sharing the data of the first file to know that the first file has been modified and the details of the modification, which helps to ensure the data consistency of the system including multiple devices.

[0032] Optionally, the first modification confirmation message includes one or more of the following: the identifier of the operating device for the first modification, the identifier of the first file on the operating device, the identifier of the device to which the first file belongs, the identifier of the first file on the device to which it belongs, and a modification confirmation instruction. The specific meanings of these parameters can be found in the description of the parameters in the aforementioned first update message. The modification confirmation instruction is used to instruct the first device to confirm the modification. For example, the modification confirmation message includes a first field, where a first value (e.g., 1) indicates that the first device has confirmed the modification.

[0033] In another possible implementation of the first aspect, the method further includes: if the first modification time is later than the target time, sending a first modification rejection message, the first modification rejection message being used to instruct the first device to reject the first modification.

[0034] For example, the first modification rejection message includes a first modification location, a first modification time, and a modification rejection indication. This rejection indication indicates that the modification is rejected. For instance, the modification rejection message includes a first field; when the first field takes a second value (e.g., 0), it indicates that the first device rejects the modification. Further, when the field takes a first value (e.g., 1), it indicates that the first device confirms the modification. Optionally, the first modification rejection message also includes one or more of the following: an identifier of the operating device, an identifier of the first file on the operating device, an identifier of the device to which the first file belongs, and an identifier of the first file on the device to which it belongs.

[0035] In the above embodiments, the first device can issue a denial of the modification to the first file, so that the second device can know whether the modification operation has been successfully accepted, which helps to ensure the data consistency of the system including multiple devices.

[0036] In another possible implementation of the first aspect, the first modification position includes an offset and a data length. The offset indicates the offset of the starting position of the first data relative to the starting position of the first file. The data length indicates the length of the first data.

[0037] In the above embodiments, the modification location can indicate the specific data location where the data update occurs, so that the first device can accurately update the content of the corresponding data block. In addition, the offset and data length are data generated when the device initiates a write operation. Using the above data as the location to indicate the data update simplifies the process and does not generate new parameters, reduces the latency of initiating update messages, and avoids the generation of redundant information.

[0038] Alternatively, the offset and data length can be replaced with the start offset and end offset corresponding to the first data, where the start offset indicates the offset of the start position of the first data relative to the start position of the first file, and the end offset indicates the offset of the end position of the first data relative to the start position of the first file.

[0039] In another possible implementation of the first aspect, the first device determines a target data block among a plurality of data blocks, including: determining the target data block based on the data location ranges corresponding to the plurality of data blocks, and the data location ranges indicated by offsets and data lengths. Each data block in the plurality of data blocks corresponds to a data location range, and the data location range indicated by the offsets and data lengths falls within the data location range corresponding to the target data block. Optionally, the data location ranges corresponding to each data block are not repeated.

[0040] In another possible implementation of the first aspect, the first modification location includes indication information of the target data block. In the above implementation, the second device can directly indicate the data block that generated the data modification, so that the first device can directly obtain the corresponding data block, reducing the workload of the first device.

[0041] In one possible implementation of the first aspect, the first device obtains first data from the second device, including the following operations: the first device obtains a target data block from the second device, the data in the target data block including the first data. Further, the first device does not obtain other data blocks besides the target data block. In some cases, for multiple data blocks of the first file, the arrangement of the multiple data blocks on the first device is consistent with the arrangement of the multiple data blocks on the second device. For example, if the first modification performed by the second device modifies the second data block on the second device, then the data block involved in the first modification determined by the first device is also the second data block.

[0042] In the above implementation, the first device obtains the data block containing the updated data in the form of data blocks, and updates the file data at the data block level, which reduces the amount of data transmitted when updating the file's data content and improves the efficiency of data synchronization.

[0043] In another possible implementation of the first aspect, the first device acquires first data from the second device, including the following operations: the first device acquires the first data from the second device based on a first modification position, the first modification position including an offset and a data length. Further, the first device does not acquire other portions of the first file besides the first data. Alternatively, the offset and data length can also be replaced with the start and end offsets of the first modification.

[0044] In the above embodiments, the first device accurately obtains updated data with detailed data location and updates the corresponding data blocks, which greatly reduces the amount of data transmitted when updating the data content of the file and improves the efficiency of data synchronization.

[0045] In another possible implementation of the first aspect, the first device is the owner device of the first file. The owner device is responsible for maintaining the complete data content of the file. For example, the owner device stores the complete data content of the file, so the first device is responsible for maintaining the latest data of the first file and can update the data content of the first file based on modifications made by other devices. By defining the owner device of the file, it is possible to determine who maintains the file data, improving data consistency in a multi-device system and enhancing the user experience.

[0046] In some cases, the file is initially created by the cluster or site that serves as the file's home device. Optionally, the file's home device can be adjusted.

[0047] In yet another possible implementation of the first aspect, the first device and the second device are located at different sites (or data centers). Furthermore, the distance between the first device and the second device is large, for example, they are located in different cities or regions.

[0048] In another possible implementation of the first aspect, the method further includes: a first device receiving modification operation information of a second modification to a first file, the modification operation information of the second modification including second data; the first device updating the first file stored on the first device based on the second data. Further, the first device determines a first data block among a plurality of data blocks based on the position of the second data in the first file. The first device updates the last modification time corresponding to the first data block in the first extended metadata of the first file to a second modification time, the second modification time indicating the operation time of the second modification.

[0049] In the above embodiment, the first file stored on the first device is modified a second time, and the second data is the data corresponding to the second modification, that is, the data updated after the second modification. In this case, the first device can update the data of the file, and at the same time update the local last modification time of the data block involved in the second modification, the above embodiment can maintain the accuracy and real-time performance of the extended metadata of the first file and improve system stability.

[0050] In some cases, when the first device modifies the first file locally, it also updates the content of the first data at the data block level and based on the modification time. Furthermore, the first device publishes update messages to other devices so that they are aware that the first file has been updated.

[0051] In another possible implementation of the first aspect, the data synchronization method further includes: a first device sending a second update message, the second update message indicating that the first device has performed a second modification on the first file, the second update message including an identifier of the device performing the second modification, an identifier of the device to which the first file belongs, a second modification location, and a second modification time, the second modification location indicating the position of the second data in the first file. Both the identifier of the device performing the second modification and the identifier of the device to which the first file belongs are identifiers of the first device.

[0052] The above implementation describes a synchronization method when the home device modifies file data. The home device can send (or publish) an update message indicating the location and time of the updated data. Furthermore, other devices can discard local data blocks based on the location and time of the data update.

[0053] In another possible implementation of the first aspect, the plurality of devices further includes a third device, on which some or all of the data of the first file is also stored. The first file stored on the third device may also be modified, in which case the third device may synchronize the modification of the first file with the first device by sending an update message.

[0054] In one example, the data synchronization method further includes: a first device obtaining a third update message from a third device, the third update message indicating that a first file stored on the third device has been modified a third time, the third update message including the third modification location and the third modification time, the third modification location indicating the location of the third data corresponding to the third modification in the first file, and the third modification time indicating the operation time of the third modification.

[0055] The first device obtains first data from the second device, including: the first device determining the data block corresponding to the third data among multiple data blocks based on the third modification location. If the first device determines that the data block corresponding to the third data conflicts with the target data block, and the first modification time is later than the third modification time, the first device obtains the first data from the second device.

[0056] In the above implementation, the modifications made to the first file by the third device and the second device involve the same data blocks. "Same" here does not necessarily mean completely identical; there may be partial overlap. However, since the third device's modification time is earlier than the second device's modification time, the first device, based on time comparison, determines that the second device's modifications should be retained, ensuring that the latest data content is stored in the first file when conflicts arise.

[0057] Optionally, the above processing can be synchronous or asynchronous. For example, the first device can merge the messages and only compare the latest modification time and the last modification time corresponding to the data block. Further, the first device determines, based on the first modification location, the second modification location, the first modification time, the second modification time, and the last modification time, that the data update of the first file by the second device is the latest version of the relevant data block.

[0058] Alternatively, the first device may process each message sequentially without pre-merging the modified messages; that is, the first device compares the time in each message with the last modified time.

[0059] In another possible implementation of the first aspect, the first device and the second device transmit messages through a message relay station. In this case, both the first device and the second device have a communication connection with the message relay station, which is used to transmit messages between the first device and the second device.

[0060] For example, the first update message is sent by the second device to the message relay station, and the first device can retrieve the first update message from the message relay station and process it asynchronously. That is, the first device retrieves the first update message from the second device, including the following steps: the first device retrieves the first update message from the second device from the message relay station.

[0061] As another example, the first modification confirmation message is sent by the first device to the message relay station, and the second device obtains the first modification confirmation message from the message relay station and processes it asynchronously. That is, the first device sends the first modification confirmation message to the message relay station, and the second device obtains the first modification confirmation message from the first device from the message relay station.

[0062] In the above implementation, the first device and the second device transmit messages through a message relay station. This eliminates the need for the first device and the second device to be aware of each other, reduces the overhead required for compatibility of different storage devices, achieves system decoupling, and improves flexibility.

[0063] In another possible implementation of the first aspect, the first device and the second device have a communication connection for transmitting messages between the first device and the second device. In the above implementations, the first device and the second device can also communicate directly via the communication connection, enabling them to directly transmit messages based on the communication connection.

[0064] Secondly, this application also provides a data synchronization method applied to a second device, for example, executed by the second device or a module within the second device. For ease of understanding, the following explanation uses the second device as the executing entity. In some cases, the second device maintains second extended metadata for the first file. This second extended metadata includes the last modification time corresponding to one or more data blocks of the first file, as recorded by the second device. It should be understood that the second extended metadata is maintained by the second device, and the last modification time of the data blocks it records is not necessarily the actual last modification time of the data blocks. For example, the actual last modification time of the data blocks is based on the last modification time recorded by the first extended metadata maintained by the first device. Furthermore, in some cases, the file's extended metadata is maintained on both the file's owner device and the file's cache device, each recording the last modification time known locally by the aforementioned devices, but this is not necessarily the actual last modification time.

[0065] The data synchronization method provided in the second aspect includes: a second device acquiring modification operation information of a first modification to a first file, the modification operation information including first data; and the second device sending a first update message to enable the first device to acquire the first data from the second device.

[0066] The first update message indicates that a first file stored on the second device has undergone a first modification. The first update message includes a first modification location and a first modification time. The first modification location indicates the position of the first data corresponding to the first modification within the first file, and the first modification time indicates the operation time of the first modification. The first modification time is later than the last modification time of the target data block, which is the data block corresponding to the first data.

[0067] Optionally, the device to which the first file belongs is a first device, which is used to manage the complete data content of the first file.

[0068] In one possible implementation of the second aspect, the method further includes: the second device updating the last modification time of the target data block recorded in the second extended metadata to the first modification time, wherein the target data block is the data block corresponding to the first data among a plurality of data blocks.

[0069] In one possible implementation of the second aspect, the first device maintains first extended metadata for the first file, which includes the last modification times corresponding to multiple data blocks recorded by the first device. The data synchronization method further includes: if the first modification time is later than the last modification time corresponding to the target data block, the second device receives a first modification confirmation message from the first device, which instructs the first device to confirm the first modification. The last modification time corresponding to the target data block is the last modification time of the target data block recorded in the second extended metadata.

[0070] Furthermore, the second device receives a data acquisition request from the first device. This data acquisition request is used to request the acquisition of first data on the second device to update the first file on the first device. At this time, the first device is able to acquire the first data and use it to update the first file stored on the first device.

[0071] In one possible implementation of the second aspect, if the first modification is not accepted, the first device may send a first modification rejection message to the second node, the first modification rejection message being used to instruct the first device to reject the first modification.

[0072] In one possible implementation of the second aspect, the method further includes: if the first modification time is later than the target time, receiving a first modification rejection message from a first device, the first modification rejection message indicating that the first device rejects the first modification. In response to the first modification rejection message, deleting the contents of a second data block or marking the second data block as old data.

[0073] For example, the first modification rejection message includes a first modification location, a first modification time, and a modification rejection indication. This rejection indication indicates that the modification is rejected. For instance, the modification rejection message includes a first field; when the first field takes a second value (e.g., 0), it indicates that the first device rejects the modification. Further, when the field takes a first value (e.g., 1), it indicates that the first device confirms the modification. Optionally, the first modification rejection message also includes one or more of the following: an identifier of the operating device, an identifier of the first file on the operating device, an identifier of the device to which the first file belongs, and an identifier of the first file on the device to which it belongs.

[0074] In some cases, the aforementioned first message can also be seen as a first rejection message for the first modification. That is, if the first device receives a conflicting modification from a later time, it means that the first modification has been rejected.

[0075] In another possible implementation of the second aspect, the method further includes: a second device acquiring a first message from a first device, the first message indicating that a first file has been modified a fourth time. The first message includes a fourth modification location and a fourth modification time, the fourth modification location indicating the position of the fourth data corresponding to the fourth modification in the first file, and the fourth modification time indicating the operation time of the fourth modification. The second device determines a second data block among a plurality of data blocks, the second data block being the data block corresponding to the fourth data. If it is determined that the second data block conflicts with a target data block, and the fourth modification time is later than the first modification time, the second device deletes the second data block or marks the second data block as old data. The first modification time is the last modification time of the second data block recorded in the second extended metadata.

[0076] In the above embodiment, the first device issues a fourth modification to the second data block. The second data block conflicts with the target data block involved in the first modification by the second device, for example, they are completely identical or partially identical, and the operation time of the fourth modification is later than the operation time of the first modification. Therefore, the data modified by the second device should be discarded. Thus, the second device deletes the content of the second data block or marks the second data block as old data, which is beneficial to maintaining the data consistency of the system.

[0077] In another possible implementation of the second aspect, the method further includes: the second device, in response to a data read request, acquiring target data from the first device and updating the last modification time of the second data block in the second extended metadata, the data read request being used to indicate reading the target data from the second device, and the target data corresponding to the second data block.

[0078] Specifically, the last modification time of the second data block in the second extended metadata is updated to the last update time of the second data block recorded by the first device. When fetching data from the second data block from the first device, the second device updates the last modification time of its local second data block to match the last modification time recorded by the first device, ensuring that the last modification times of the two devices are aligned.

[0079] The target data can be exactly the same as the second data block, or the target data can partially overlap with the data in the second data block.

[0080] In the above embodiments, if the first device has not received the first modification, for example, if it has received another later modification targeting the target data block, the second device can obtain new data from the first device and update the second data block and the last modification time corresponding to the second data block.

[0081] In another possible implementation of the second aspect, the method further includes: the second device generating a first tag message for the first data, the first tag message being used to mark that the first data will not be evicted before the first device obtains the first data from the second device. For example, the second device generates a write message for a target data block in the second device, the write message being able to protect the target data block from being evicted.

[0082] In another possible implementation of the second aspect, the method further includes: after receiving a first modification confirmation message or a first message from the first device, the second device deletes the first tag message. The first modification confirmation message instructs the first device to confirm the first modification, and the first message instructs the first device to confirm a modification whose operation time is later than the operation time of the first modification; see the description in the foregoing embodiments for details. In the above embodiments, after the first device accepts the modification from the second device, the first device does not need to protect the target data block from being evicted; in this case, the first tag message can be deleted.

[0083] In some cases, after the first device accepts the modification from the second device, the second device also generates read information for the first data or the data block containing the first data.

[0084] In some cases, even if the first device has not received the first modification, such as if it has received other, later modifications to the target data block, the second device may also delete the aforementioned first tag message.

[0085] In another possible implementation of the second aspect, the first modification position includes an offset and a data length, the offset indicating the offset of the starting position of the first data relative to the starting position of the first file, and the data length indicating the data length of the first data.

[0086] In another possible implementation of the second aspect, the first modification location includes indication information of the target data block.

[0087] Thirdly, this application also provides a data synchronization device, which includes a communication module and a processing module. The communication module is used to receive and / or send messages, and also to perform other operations that require communication with the outside world, such as receiving file-related input / output (I / O) operations, such as read and / or write operations. The processing module is used to process information, such as performing one or more operations such as comparison, decision-making, updating, generating, determining, writing, or reading.

[0088] In one possible implementation, the data synchronization device is used to implement the method described in the first aspect or any possible implementation of the first aspect.

[0089] In another possible implementation, the data synchronization device is used to implement the method described in the second aspect or any possible implementation of the second aspect.

[0090] In some cases, a data synchronization device is included within a device, which may also include a storage unit or be connected to a storage unit. The data synchronization device enables synchronization between data stored in the device and data stored on other devices.

[0091] Fourthly, this application also provides an apparatus comprising a processor and a memory, and the apparatus further comprising a storage unit or connected to the storage unit. The storage unit is used to store data of a file system and may be referred to as a storage disk, such as a hard disk, solid-state drive, or partition.

[0092] The aforementioned memory is used to store computer instructions. The processor is used to invoke the computer instructions stored in the memory to implement the data storage method described in the first aspect or any possible embodiment of the first aspect. And / or, the processor is used to invoke the computer instructions stored in the memory to implement the data storage method described in the second aspect or any possible embodiment of the second aspect.

[0093] Fifthly, this application also provides a data synchronization device, which includes a processor and a memory, wherein the memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to cause a computing device to implement the method described in the first aspect or any possible implementation of the first aspect, and / or to implement the method described in the second aspect or any possible implementation of the second aspect.

[0094] The data synchronization device can be installed within the device, which also includes a storage unit or is connected to the storage unit. In the latter case, the data synchronization device within the device can be connected to the storage unit. This connection can include a direct connection via a connection line or a network connection; for example, the data synchronization device and the storage unit can be connected via a switch.

[0095] Sixthly, this application also provides a data synchronization system, which includes multiple devices, including a first device and a second device. The first device is used to implement the method described in the first aspect or any possible implementation thereof, and the second device is used to implement the method described in the second aspect or any possible implementation thereof.

[0096] In some cases, the first device stores file system data, and the first device can share the file system data and metadata externally. Note that in this embodiment, the extended metadata of the file may not be shared with external devices. The second device can obtain the file system metadata and construct a file view of the file system. The second device can also load data of some files in the file system from the first device to its local machine. Further, the file system includes a first file, and the data of the first file is managed in the form of data blocks.

[0097] In a seventh aspect, this application also provides a computer-readable storage medium for storing instructions or computer programs that, when executed, implement the methods described in the first aspect or any possible implementation of the first aspect, and / or implement the methods described in the second aspect or any possible implementation of the second aspect.

[0098] Eighthly, this application also provides a computer program product, which includes computer program instructions that, when executed, implement the method described in the first aspect or any possible implementation of the first aspect, and / or implement the method described in the second aspect or any possible implementation of the second aspect.

[0099] Computer program instructions can include one or more of programs, instructions, etc.

[0100] The beneficial effects of the solutions in aspects two through eight of this application can be found in the beneficial effects of aspect one, and will not be repeated here. Attached Figure Description

[0101] Figure 1 is a schematic diagram of the architecture of a data synchronization system provided in an embodiment of this application;

[0102] Figure 2 is a schematic diagram of the structure of a first device provided in an embodiment of this application;

[0103] Figure 3 is a schematic diagram of the architecture of another storage system provided in an embodiment of this application;

[0104] Figure 4 is a schematic diagram of file data and extended metadata provided in an embodiment of this application;

[0105] Figure 5 is a flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0106] Figure 6 is a schematic diagram of the operation scenario of a data synchronization method provided in an embodiment of this application;

[0107] Figure 7 is a schematic diagram of an update message format provided in an embodiment of this application;

[0108] Figure 8 is a schematic diagram of the format of a modification confirmation message provided in an embodiment of this application;

[0109] Figure 9 is a schematic diagram of the operation scenario of another data synchronization method provided in the embodiments of this application;

[0110] Figure 10 is a schematic diagram of a data synchronization device provided in an embodiment of this application;

[0111] Figure 11 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0112] The following section will introduce some of the terms used in the embodiments of this application.

[0113] 1. File System: A file system is a method (i.e., a method of organizing files on a storage disk) and a data structure used to identify files on a storage disk. The main function of a file system is to allow users to easily read and write files. For example, a user provides the file system with an identifier for a specified file (such as the file name or the file path), and the file system can access the data of that file.

[0114] File systems can include one or more of the following types: network file system (NFS), file system based on the server message block (SMB) protocol, common internet file system (CIFS), simple storage service (S3), object storage service (OBS), distributed file system, etc. Further, distributed file systems include, for example, Lustre file system, GPFS, HDFS, Ceph file system, or Swift file system, etc.

[0115] The file system involved in this application is a system with a tree-like hierarchical structure that provides access and retrieval services for multiple files. In some scenarios, systems with similar characteristics may not necessarily be called file systems, but for ease of description, they are uniformly referred to as file systems in this document. For example, some object systems, when storing objects, also have a tree-like hierarchical structure among multiple objects, which also fall within the scope of "file system" in the embodiments of this application.

[0116] 2. Files, Data, and Metadata: A file, or computer file, is a collection of information. A file includes data and metadata. Data refers to the file's content. Metadata describes the file's information, such as its name, size, and type. For example, a file system might include a file named "001.png". The metadata for this file could describe its name ("001.png"), type, size, location, creator, creation time, or permissions.

[0117] 3. Global File System (GFS): Also known as a federated file system, it unites multiple file systems and provides a unified view of them, i.e., a global file view. Users can load this global file view on any device and access file data on other devices within the GFS framework. GFS provides a unified file system namespace and a global file view for applications across data processing systems (e.g., across data centers). Business applications do not need to be aware of the data's location; they can access the unified global file view from any device connected to GFS. When an application reads or writes a non-local file, GFS loads the data from the remote location onto the local device. At the application level, there is no distinction between local and non-local data; response speed depends on the network bandwidth and latency between devices.

[0118] For example, in some solutions, a user deploys NFS at site A and S3 at site B. Both sites can access GFS. Devices and services connected to GFS can then view a unified global file view combining NFS and S3 from any site, such as site C, and can read data from both local and non-local files. From the user's perspective, multiple file systems form a unified file system, making it possible to manage data across multiple devices in a consistent manner. This greatly simplifies file management across multiple devices and enables file sharing, flow, and fast access across them.

[0119] 4. Ownership: The metadata of a file may contain ownership information, which specifies the device to which the file belongs, such as the device to which the file belongs. The file's ownership device is used to manage the file's data, including but not limited to maintaining the file's latest complete data content, publishing data changes when the file's data changes, publishing metadata changes when the file's metadata changes, or distributing data (e.g., returning data to applications that request data).

[0120] File ownership can be determined using predefined rules. In some cases, the file's ownership device is the cluster or site that was first created. In other cases, the file's ownership device can be updated, for example, by a user or automatically based on predefined rules.

[0121] The above is an explanation that can be applied to the embodiments described below.

[0122] As user business data grows in scale, more and more users are using storage systems that include multiple storage devices to store their data. To adapt to user needs, storage systems that support the interconnection of multiple storage devices, such as global file systems, cloud drives, and cloud desktops, are constantly emerging. Taking a global file system as an example, user business data is stored jointly by multiple storage devices, and the data from these multiple storage devices is logically combined to form a unified file system, namely a global file system. Devices and services connected to the global file system can view the overall file view of the global file system, and data in files within the global file system can flow between multiple storage devices. In addition, the global file system also supports accessing global file data from any device or service, and loading remote data to the local machine.

[0123] In storage systems with multiple storage devices, synchronizing file system data across these devices becomes a challenge, especially when the same file is modified on multiple devices, which can lead to file modification conflicts. Resolving these conflicts is a crucial aspect of the data synchronization process.

[0124] Currently, the method to resolve file modification conflicts is to split conflicting files into two independent files. For example, a storage system might include storage device S1, S2, and S3. File data can flow freely within the global file system, so data consistency must be maintained across the storage system. In a storage system with multiple storage devices, multiple devices may modify the same file, causing file modification conflicts. For instance, if both storage device S1 and storage system S2 modify file File1, to preserve the modifications from both devices, the storage system might split the file into two files. One file might still be named File1, and the other might be named File1-Conflicting Version 1 (this naming convention is just an example). In this method, the number of files increases continuously due to the constant occurrence of conflicts, leading to reduced file system storage space utilization, increased redundancy, and increased costs. Furthermore, due to the lack of an effective merging mechanism, users will see multiple redundant files in the file view, requiring them to manually review and delete unnecessary versions, and potentially rename them, resulting in a poor user experience.

[0125] In some alternative solutions, the storage system directly overwrites the older, more complete data with the latest modified file. Continuing with the file system example above, storage devices O1, S1, and S2 all store data for File1. Storage device S2 modifies File1 at time T2, while storage device S1 modifies File1 at time T1 (which is later than T2). In this case, the storage system retains the modified File1 data from storage device S1 as the complete version of File1, while the modification made by storage device S2 is discarded. This solution can prevent redundant files, but it undoubtedly leads to frequent loss of modifications made to files by some storage devices, does not support collaborative updates of file data, and results in a poor user experience.

[0126] In view of this, this application provides a data synchronization method and related products that can merge modifications to files by storage devices at the data block level, support collaborative updates of file data in the file system by multiple storage devices, and realize file synchronization between multiple storage systems. Furthermore, the solution provided in this application also solves the problem of additional storage space occupation caused by conflicting file modifications, significantly improving the user experience.

[0127] The following describes some possible system architectures to which this application applies. It should be understood that the system architectures provided below are exemplary systems listed to facilitate understanding of the application scenarios of this solution. Those skilled in the art should understand that with the emergence of new business scenarios and new system architectures, this application is also applicable to solving similar technical problems.

[0128] Please refer to Figure 1. Figure 1 is a schematic diagram of the architecture of a data synchronization system provided in an embodiment of this application. The data synchronization system includes a first device 10 and a second device 20, and optionally also includes a third device 30. Wherein:

[0129] Data is stored on the first device 10, the second device 20, and the third device 30. For example, the first device 10 stores the data of file File1, while the second device 20 and the third device 30 store some or all of the data of file File1. The aforementioned devices may include a single device or a cluster of multiple devices, such as a data center (DC) or storage servers. In some solutions, one or more devices can form a site.

[0130] In some cases, files are stored on a device, including two scenarios: Scenario 1, the device is a storage device, including storage units, and the file is stored in the device's storage units. Scenario 2, the device is a computing device, and the file is stored on a storage unit accessible to the device or managed by the device. Optionally, the aforementioned devices, such as the first device 10, the second device 20, and the third device 30, include storage units, or the aforementioned devices are connected to storage units. The first device 10 is described below as an example. The first device 10 includes storage units (referred to as the first storage unit for ease of distinction), and / or the first device 10 is connected to the first storage unit. The first storage unit provides storage space for the first device 10, and the data stored in the first storage unit can be considered as data stored on the first device 10. It should be understood that the storage unit here represents a module with storage space, and in some scenarios, it may also be called a storage disk. For example, a storage unit may include one or more of the following: mechanical hard disk, solid-state drive, hybrid hard disk, etc. Alternatively, the storage unit may also be virtual, such as a virtual storage pool. It should be understood that the number of storage units in a device is not strictly limited in this application.

[0131] In one possible implementation, in the data synchronization system, the first device 10 stores data from the file system FS1, where file File1 is a file within the file system FS1. Furthermore, the first device 10 also stores metadata for the file system FS1. Optionally, the first device 10 is the owner device of the file system FS1 (or file File1), or a data source.

[0132] The aforementioned device also possesses communication and computing capabilities. The communication capability enables the device to send and / or receive messages, while the computing capability enables it to process information. As a possible example, the device includes a data synchronization unit, which possesses both communication and computing capabilities. Here, the data synchronization unit represents a device or module with both communication and computing capabilities. As a possible implementation, the data synchronization unit can send and / or receive messages, and perform one or more operations such as judgment, acquisition, updating, calculation, control, and decision-making. For ease of distinction, the data synchronization unit in the first device 10 is referred to as the first data synchronization unit, and the modules in the other devices are also given corresponding distinctive names. It should be noted that the use of "data synchronization unit" to represent a module within the device with computing and communication capabilities is primarily for ease of description and is not intended as a strict limitation on the device's structure. In some scenarios, the data synchronization unit can be divided into more modules, and these modules can even be set up separately, for example, on different entities.

[0133] In one possible implementation, the data synchronization device can be implemented in software, in hardware, or in a combination of both.

[0134] As an example of a data synchronization device implemented in hardware, the data synchronization device can be a computing device or a module within a computing device. The first device 10 is described below as an example; the second device 20 and the third device can be deduced similarly. Referring to Figure 2, the first device 10 includes a computing device and a storage unit. The computing device includes one or more of the following: a controller, a processor, or a server. The controller includes, but is not limited to, a storage controller (e.g., a memory controller, a hard disk controller, an integrated drive, an electronic controller, a disk array controller, etc.), a combinational logic controller, a hardwired controller, etc. The processor includes, but is not limited to, a central processing unit, a graphics processor, an artificial intelligence processor, a microprocessor, or a programmable gate array (FPGA). Furthermore, in some scenarios, because the controller also has computing capabilities and / or can execute instructions, it can also be considered a processor. The server includes, but is not limited to, a general-purpose computer, a storage server, a cloud server, or a blade server. When the functionality of the computing device is implemented by a server, the number of servers it contains can be one or more (e.g., a server cluster).

[0135] Optionally, the computing device and storage unit in the first device 10 are integrated. As an example of the integrated computing device and storage disk configuration, the first device 10 is a storage system with integrated disk controller. The first device 10 includes a controller (the number of which can be one or more), and the controller is connected to the storage unit (e.g., hard disk) via a bus.

[0136] Alternatively, the computing device and storage unit can be configured independently. Of course, regardless of whether the computing device and storage unit are configured independently or integrated into the same device, they are interconnected, and the connection method can be a bus or a network. The network can be, for example, a wired network, a wireless network, or a combination of wired and wireless networks. For example, they can be connected via a network cable or via a switch.

[0137] The above describes the hardware implementation of a data synchronization device. Below are examples of data synchronization devices implemented in software. In some examples, the functions of the data synchronization device can be implemented through software functional units. For example, the data synchronization device may include computing instances such as virtual machines, containers, and cloud computing. A virtual machine is a computer system with complete hardware system functionality simulated by software and running in an isolated environment. A container is an isolated environment obtained by packaging applications and their dependencies. The cloud is a software platform using application virtualization technology, enabling one or more software programs or applications to be developed and run in an independent virtualized environment. Optionally, the cloud may be deployed on a public cloud, private cloud, or hybrid cloud. In still other examples, the data synchronization device may include computer-executable content such as computer instructions, computer programs, or computer code running on a computing instance. The computing instance may include at least one of the following: a physical host, a virtual machine, a container, or other device with instruction execution capabilities.

[0138] In the aforementioned data synchronization system, devices connected to the system can communicate using messages. This communication includes direct communication or indirect communication based on message relay stations. For example, the first device 10 and the second device 20 can communicate through a message relay station. Alternatively, the first device 10 and the second device 20 may have a communication connection, allowing them to transmit messages using that connection.

[0139] Optionally, in the data synchronization system shown in Figure 1, the structures of multiple devices can be the same or different. For example, some devices adopt an integrated storage and control architecture, while some devices can adopt an architecture combining virtual machines and virtual storage pools.

[0140] Alternatively, the methods used by multiple devices to organize files can be different; for example, some devices may use NFS as the file system protocol, while others may use OBS.

[0141] In some cases, the first device 10 and the second device 20 are located at different sites (or data centers). Furthermore, the first device 10 and the second device 20 are far apart, for example, located in different cities or regions.

[0142] It should be understood that the number of devices, their structure, and the data stored in the data synchronization system shown in Figure 1 are merely examples. In a real implementation, the data synchronization system may include more devices, and the data stored in each device may have different designs.

[0143] In a data synchronization system involving multiple devices, data from files stored on the devices can flow within the system. For example, a file named File1 originally stored on the first device 10 can also be stored on the second device 20 and the third device 30. Therefore, the data within the data synchronization system needs to remain consistent.

[0144] In some cases, multiple devices are interconnected based on data flow. One device can access data stored on other devices and load it into local storage. For example, the first device 10 stores a file system FS1, which includes multiple files. The second device 20 and the third device 30 can access data from one or more files in the file system FS1 and load some of the data into local storage. For example, the file system FS1 includes a file File1. The first device 10 stores the data of file File1, while the second device 20 and the third device 30 can obtain some or all of the data of file File1 and store it locally. Therefore, the second device 20 and the third device 30 also store the data of file File1. Referring to Figure 1, devices in a data synchronization system can be connected through a data channel. Data in the data synchronization system flows through this data channel. Of course, this data channel is a virtual channel, for example, it can be implemented through a data relay station.

[0145] In some cases, different devices may have different roles for the same file. For example, for file File1, the first device 10 is its owner device, or data owner, while the second device 20 and the third device 30 are cache devices, or data cachers.

[0146] In some cases, files in a multi-data synchronization system can have multiple identifiers. As a possible design, the types of file identifiers include global identifiers and / or local identifiers. Global identifiers are used to uniquely identify a file within the data synchronization system; for example, they can be unique IDs, numbers, or file names. The following explanation uses a universally unique identifier (UUID) as an example. Local identifiers are used to uniquely identify a file on one or more devices (e.g., a site). Local identifiers also include the file's UUID on the local device.

[0147] In some possible implementations, the device accessing the data synchronization system is capable of handling read and / or write requests for files within the data synchronization system. For example, a user of the second device 20 can request to read part or all of the data in File1. If the data in File1 is stored on the second device, the second device 20 can directly respond to the read request. Otherwise, the second device 20 can obtain part or all of the data in File1 from the first device 10, store it on the second device 20, and respond to the user's read request. For write requests, the user can initiate modifications to the file on the device; for example, the user can modify the data in File1 on the first device 10 or on the second device 20.

[0148] In one possible implementation, the first device 10 can also share the metadata of the locally stored file system FS1. Other devices can obtain the metadata of the file system FS1 and construct a file view of the file system FS1 based on the metadata, so that users can view and operate on files. That is, any device in the data synchronization system can view the file view of the file system FS1 that is consistent with that of the first device 10. For example, referring to Figure 1, the second device 20 and the third device 30 can both obtain the metadata of the file system FS1.

[0149] Furthermore, other devices can also share the metadata of their locally stored file systems. In some cases, a device can combine its local file system with file systems on other devices, constructing a global file view based on the metadata of the local file system and the file systems on other devices. See Figure 3, where G_FS1, G_FS2, and G_FS3 are the file systems in the first device 10, the second device 20, and the third device 30, respectively. Black ellipses represent files belonging to the local device, and white ellipses represent files belonging to other devices. The files stored on the three devices form a global file system, for example, by combining them through a global service. Furthermore, the metadata among the three devices is shared and synchronized. Furthermore, the data among the three devices also needs to remain consistent. For each device, if the data of a file is modified locally, it needs to notify other devices of the modification so that different devices can eventually access the latest data.

[0150] Optionally, under the architecture of the global file system, the first device 10, the second device 20 and the third device 30 can all handle read requests and / or write requests to the global file system.

[0151] The architecture of the data synchronization system used in the embodiments of this application has been described above. Some designs in the embodiments of this application are described below.

[0152] In this embodiment, file data can be managed in the form of data blocks. Referring to part (a) of Figure 4, the data of the first file File1 is stored in the form of multiple data blocks, such as data block 1 (abbreviated as block 1), block 2, block 3, etc. It should be understood that these multiple data blocks are abstracted at the management level, and each data block corresponds to a range of locations within the data of the first file. That is, the division of data blocks can be logical and does not necessarily represent the data being stored in the actual physical form of multiple blocks.

[0153] In some cases, multiple data blocks occupy the same size; for example, in the first file shown in Figure 4, each data block occupies a size of 4 megabytes (MB). In other cases, the size of each data block in the multiple data blocks is not equal; for example, the size of the data blocks in the file is predefined, such as cycling between 4MB and 2MB.

[0154] In this embodiment of the application, one or more devices may store extended metadata of a file. The extended metadata of the file includes the last modification time of the data blocks of the first file, and optionally also includes the identifier of the first file. The last modification time indicates a moment or a unit of time, and may include a timestamp or a version number.

[0155] Taking the last modified time as a timestamp as an example, as shown in part (b) of Figure 4, the extended metadata of File1 includes the identifier of File1 and the corresponding last modified time for each data block of File1, which is indicated by a local timestamp. For example, the extended metadata of File1 includes the local timestamp corresponding to block 1 (value T_b1), the local timestamp corresponding to block 2 (value T_b2), and the local timestamp corresponding to block 3 (value T_b3). The last modified time is used to indicate the creation time or the most recent update time of the corresponding data block stored on the device where the extended metadata resides. Optionally, the file identifier in the extended metadata can be a global identifier or a local identifier of the file.

[0156] In some cases, extended metadata for a file is maintained on both the file's owner device and its cache device, each recording the last modification time as known locally on those devices, but not necessarily the actual last modification time. Taking a first file as an example, data for the first file is stored on both a first device 10 and a second device 20. The first device 10 is the owner device for the first file, and the second device 20 is the cache device for the first file. The first device 10 maintains the first extended metadata for the first file, and the second device maintains the second extended metadata for the first file. The first extended metadata includes the last modification times corresponding to multiple data blocks recorded by the first device, and the second extended metadata includes the last modification times corresponding to one or more data blocks of the first file recorded by the second device.

[0157] In some cases, a file's extended metadata may not be shared with external devices. In some implementations, the file's extended metadata is an abstract set of information, whose specific parameters may be distributed across the file's metadata or other attribute data.

[0158] The above describes the multiple data blocks of a file and its extended metadata. For ease of description, the following explanation uses the first file as an example. The extended metadata of the first file is stored at least on the first device 10. Of course, the extended metadata of the first file can also be stored on devices such as the second device 20 and the third device 30 that store the data (partial or all of the data) of the first file.

[0159] In this embodiment, the data of the first file is stored in the first device 10 and the second device 20, and optionally also in the third device 30. The first device 10 also maintains the first extended metadata of the first file. In a system including multiple devices, the first file stored on the devices may be modified. Taking the first file stored on the second device 20 as an example, the second device 20 sends a first update message to the first device. The update message can indicate the location of the data corresponding to the first modification and the operation time of the first modification. The first device determines the target data block involved in the first modification based on the location of the data corresponding to the modification, and obtains the last modification time of the target data block based on the first extended metadata. The first device compares the last modification time of the target data block with the operation time of the first modification. If the operation time of the first modification is later than the last modification time of the target data block, it indicates that the first modification is a newer modification. Then, the first device obtains the data obtained from the first modification from the second device and updates the first file stored in the first device, so that the data modification of the first file is successful.

[0160] As can be seen, the first device maintains the last modification time of multiple data blocks corresponding to the first file. When a modification occurs, the comparison between the modification operation time and the last modification time is used as the basis for updating the data block content, enabling data modifications to the first file by other devices to be merged at the data block level. Similarly, other devices can also send update messages when making modifications, and the first device also retains the latest modification based on time comparison. In summary, this application can support collaborative updates of file data in the file system by multiple devices, realizing file synchronization within a system with multiple storage devices.

[0161] In some cases, multiple devices may modify the same file, resulting in a file modification conflict. Given the varying degrees of conflict, two possible scenarios exist:

[0162] First, regarding modifications to different data blocks of the same file, for example, if a second device 20 modifies data block 1 of a first file, while a third device 30 modifies data block 2 of the same file, there is no conflict at the data block level because time comparisons and data updates are performed at the data block level. For instance, the first device 10 updates data block 1 with the modifications from the second device 20, and updates data block 2 with the modifications from the third device 30, ensuring that modifications from multiple devices are preserved. Managing file data at the data block level avoids the problem of lost modifications when file modification conflicts occur, thus improving the user experience.

[0163] Second, regarding modifications to the same data block of the same file, for example, if the second device 20 modifies data block 1 of the first file at time T1, and the third device 30 modifies data block 1 of the first file at time T2, since the time comparison and data update are performed at the data block granularity, after the first device processes the above modifications, the content of the last retained data block B1 is the modification corresponding to the latest time among time T1, time T2, and the last local modification time. That is, the data block of the first file stored in the first device contains the latest data content. In other words, even if there are modifications at the data block level, the solution using the embodiments of this application will not generate new files, avoiding increased costs.

[0164] In summary, the embodiments of this application can merge modifications to files by devices at the data block level, support collaborative updates of file data in the file system by multiple devices, realize file synchronization between multiple devices, solve the problem of extra storage space occupation caused by conflicting modifications of files, and improve the user experience.

[0165] Furthermore, the second device also maintains the second extended metadata of the first file. Based on the second extended metadata and the received messages, the second device can determine whether the data of the second file stored in the second device is the latest data.

[0166] The method provided by the embodiments of this application is described below. Please refer to Figure 5, which is a flowchart illustrating a data synchronization method provided by an embodiment of this application. Optionally, this method can be applied to a data synchronization system including multiple devices, such as the data synchronization system shown in Figure 1 or Figure 3 above. The data synchronization method shown in Figure 5 may include one or more steps S501 to S506. It should be understood that, for ease of description, the steps S501 to S506 are described in this way, and it is not intended to limit the execution to the above order. This application embodiment does not limit the executing entity, the order of execution, the execution time, the number of executions, etc. of the above one or more steps. Steps S501 to S506 are as follows:

[0167] S501, the second device obtains modification operation information of the first modification to the first file.

[0168] The second device stores some or all of the data of the first file. Optionally, the data of the first file can be cached only on the second device, or the second device can persistently store the data of the first file.

[0169] In some cases, a first modification to a first file updates the data of the first file, which can be considered as a write operation performed on the first file by the second device. For example, the second device acquires the user's write operation, which can modify the data of the first file. The modification information of the first modification includes the first data; for example, for a write operation, the write operation includes newly written data, and the first data is the newly written data. In short, the first data is the data corresponding to the first modification. Referring to part (a) of Figure 6, the first modification can update the data between 4.13M and 6.13M of the File1 file, and the operation time of the first modification is T1; therefore, the first data is the data between 4.13M and 6.13M.

[0170] In some cases, the first modification is an edit of existing data, i.e., replacing existing data. Alternatively, the first data can be newly generated data, i.e., newly added data.

[0171] In some possible implementations, the second device maintains second extended metadata for the first file. This second extended metadata includes the last modification times of one or more data blocks in the first file, recorded by the second device. After the first file undergoes a first modification, the second device updates the last modification time of the target data block recorded in the second extended metadata to the first modification time. The target data block is the data block corresponding to the first data block among multiple data blocks. For example, if the last modification time of the target data block is T1 in the second extended metadata, then after the first modification is performed, the second device modifies the last modification time of the target data block to T1.

[0172] In some solutions, step S501 is an optional step, meaning that updating the data may not necessarily depend on the existence of modification operation information. For example, in cases of system rollback, special service calls, etc., file modification operations may occur, resulting in updated data and corresponding operation timestamps.

[0173] S502, the second device sends a first update message to the first device.

[0174] The first update message indicates that a first file stored on the second device has undergone a first modification. The first update message includes a first modification location and a first modification time. The first modification location indicates the position of the first data corresponding to the first modification in the first file, and the first modification time indicates the time of the first modification operation. The following sections describe some possible implementations of the modification location and modification time indication information.

[0175] In some cases, changing the position can be achieved in the following two ways:

[0176] In the first implementation method, the first modification location includes an offset and a data length. The offset indicates the position of the first data relative to the beginning of the first file. The data length indicates the length of the first data. For example, referring to Figure 6, if the first data is the segment from 4.13M to 6.13M, then the offset is 4.13M (or an address offset), and the data length is 2M (or an address length).

[0177] Optionally, the first modification position may also include a start offset and an end offset. For example, if the first data is the data segment from 4.13M to 6.13M, then the start offset is 4.13M and the end offset is 6.13M.

[0178] In the second implementation method, the first modification location includes the indication information of the target data block. Referring to Figure 6, the data block involved in the data segment from 4.13M to 6.13M is block 2. The first modification location may include information indicating block 2, such as the sequence number of block 2 (i.e., 2) or the ID (or number) of block 2.

[0179] In some cases, for multiple data blocks of a first file, the arrangement of the multiple data blocks on the first device is consistent with the arrangement of the multiple data blocks on the second device. For example, if the first modification performed by the second device modifies the second data block, then the data block involved in the first modification determined by the first device is also the second data block.

[0180] The two implementation methods described above are merely examples, and other implementation methods may exist in specific implementations. For example, the first modification position indicates the start position and end position of the modification, such as indicating positions 4.13M and 6.13M. For ease of description, the aforementioned implementation method one of some embodiments of this application is used as an example to illustrate the modification position.

[0181] In some cases, the modification time can be achieved through the following two designs:

[0182] Design 1: The modification time includes a timestamp. A timestamp is a unique numerical marker generated at a specific point in time to identify the exact moment an event occurred. For example, the first modification time is the first timestamp, which indicates that the operation time for the first modification is T1.

[0183] In some cases, timestamps are used to mark time across multiple devices, requiring time synchronization between these devices. For example, multiple devices may connect to a Network Time Protocol (NTP) server.

[0184] Design 2 includes a version number in the modification time. The version number is a monotonically increasing or monotonically decreasing identifier used to indicate the time of modification. For example, after initiating the first modification, the first device retrieves a first version number from the management device. This first version number indicates the time of the first modification. Devices can identify the time of modification by comparing the version numbers corresponding to multiple modifications. The management device is a predefined device, such as a message relay station.

[0185] In some possible implementations, the update message may also include one or more of the following: an identifier of the operating device, an identifier of the first file on the operating device, an identifier of the device to which the first file belongs, and an identifier of the first file on the device to which it belongs. Here, the operating device refers to the device that performs the modification, or is also called the modification source, modification operation site, etc., so that the first device can obtain the modified data from the device that performs the modification. As mentioned above, if the device performing the first modification is the second device, then the first update message may also include the identifier of the second device to indicate that the device performing the first modification is the second device.

[0186] The identifier of the first file on the operating device is used to identify the first file on the operating device, so that other devices can retrieve the modified file data from the operating device. For example, if the second device performs the first modification, the operating device in the first update information message is the second device, and the identifier of the first file on the operating device is the identifier of the first file on the second device. In some cases, each device maintains both a local identifier and an identifier for the first file in the global file system. This facilitates the management of the first file's data content while maintaining compatibility with the respective namespaces of each device. Other devices can retrieve the first file's data from the operating device using the identifier of the first file on the operating device.

[0187] The identifier of the device that owns the first document is used to identify the device that owns the first document, so that the device that owns the first document can manage the data of the first document, such as receiving messages related to the first document. In some cases, the device that owns the first document is used to process update messages related to the first document, and carrying the identifier of the device that owns the first document in the update message helps the device that owns the first document maintain the data content of the first document.

[0188] The identifier of the first file on the home device is used to identify the first file on the home device so that the home device can locate the file's data and / or the file's extended metadata.

[0189] The above describes the contents of the first storage update message. For ease of understanding, Figure 7 provides a possible format diagram of the update message. The update message includes one or more of the following information: operation site identifier, operation site file identifier, offset, length, operation timestamp, and home site identifier or home site file identifier. The operation site identifier indicates the identifier of the site initiating the data operation. For example, when the second device initiates data modification, the value of the operation site identifier field is the identifier of the second device. The operation site file identifier indicates the local identifier of the modified file at the operation site. For example, the identifier of the first file on the second device (as mentioned above). The offset and data length indicate the location of the data modification. Referring to Figure 6, the offset is, for example, 4.13M (or an address offset), and the data length is, for example, 2M (or an address length), thus indicating that the data modified after the first modification corresponds to the data segment from 4.13M to 6.13M. The operation timestamp indicates the operation time of the modification, which can be considered as the aforementioned first modification time. The home site identifier is used to indicate the site to which the file belongs. For example, if the home device of the first file is the first device, then the home site identifier is the identifier of the first device. The home site file identifier is used to describe the unique identifier of the modified file within its home site.

[0190] It should be understood that the names of messages, information, devices, and modules in this application are merely examples, and other names may be designed in specific implementations. For example, an update message may be called a modification message or a modification log.

[0191] In one possible scenario, the first update message is sent to a message relay station, and the first device receives the first update message from the message relay station. In this case, both the first and second devices have a communication connection with the message relay station; "having a communication connection" means being able to send and receive messages. That is, the message relay station is used to transmit messages between the first and second devices. For example, the first update message is sent by the second device to the message relay station, while the first device can retrieve the first update message from the message relay station and process it asynchronously.

[0192] In other possible scenarios, the first device and the second device have a communication connection used to transmit messages between them. In the above embodiments, the first device and the second device can also communicate directly via the communication connection, enabling them to directly transmit messages based on that connection.

[0193] In some possible implementations, the first device is the device that owns the first file. That is, when the data of the file is modified, a device other than the file's owning device can send an update message to the file's owning device, enabling the owning device to detect the data update and perform corresponding operations to maintain data consistency.

[0194] Step S503: The first device receives a first update message from the second device.

[0195] As described in step S502, the first update message can be obtained from the message relay station or received directly from the second device.

[0196] Step S504: The first device determines the target data block among multiple data blocks.

[0197] The data in the first file is managed in the form of multiple data blocks. The target data block is the data block corresponding to the first data, that is, the data block involved in the first modification. Referring to part (b) of Figure 6, the data modified in the first modification is the data segment from 4.13M to 6.13M, so the target data block is block 2.

[0198] Furthermore, the first device determines the target data block among multiple data blocks based on the first modified position.

[0199] As a possible example, the first modification location includes an offset and a data length. The first device determines the target data block based on the data location range corresponding to multiple data blocks, as well as the data location range indicated by the offset and data length. Each of the multiple data blocks corresponds to a data location range, and the data location range indicated by the offset and data length falls within the data location range corresponding to the target data block. As shown in Figure 3, the data location range corresponding to block 1 is 0 to 4M, while the data location range corresponding to block 2 is 4 to 8M. The first modification location indicates an offset of 4.13M and a length of 2M, which falls within the data location range corresponding to block 2. Therefore, the target data block is block 2.

[0200] Optionally, the range of data locations corresponding to each data block is not repeated.

[0201] As another possible example, the first modification location includes the identifier of the target data block, such as the identifier of block 2. This identifier can be an ID, sequence number, or serial number. The first device can determine the target data block based on its identifier, such as if the target data block includes block 2.

[0202] Step S505: If the first modification time is later than the last modification time corresponding to the target data block, the first device acquires the first data.

[0203] The last modification time corresponding to the target data block is the same as the last modification time of the target data block. The first modification time indicates the operation time of the first modification; that is, the first modification time is the operation time of the first modification. The first device compares its local last modification time with the operation time of the first modification performed by the second device. If the operation time of the first modification is later than the local last modification time of the target data block, it indicates that the first modification is a new modification, and the first data is obtained from the second device.

[0204] The first device stores extended metadata for the first file, including the last modification time of the data blocks of the first file. Referring to Figure 6, the target data block is block 2. The extended metadata of File1 includes the last modification time of block 2, which indicates T_b2, while the first modification time can indicate T1. Since T1 is greater than T_b2, it indicates that the modification of the first file by the second device is newer than the data version of the first file stored by the first device. Therefore, the first device retrieves the first data from the second device. Similarly, other devices can also update the content of a file's data block dimension by sending update messages when making modifications, and the first device also retains the latest modification based on timestamp comparison. In summary, this application can support collaborative updates of file data in a file system by multiple devices, realizing file synchronization between multiple data synchronization systems.

[0205] Alternatively, the first data can be obtained in the following two ways:

[0206] Method 1: The first device obtains the target data block from the second device. Furthermore, the first device does not obtain other data blocks unrelated to the first modification. Referring to Figure 6, the first device can obtain the data of block 2 from the second device without needing to obtain the data of blocks 1 and 3.

[0207] Method 2: The first modification location includes an offset and a data length. Based on the first modification location, the first device obtains the first data. Furthermore, the first device does not obtain any other data in the first file besides the first data.

[0208] The two acquisition methods above are just examples; other designs are possible in actual implementations.

[0209] In one possible implementation, the first file is a large file, i.e., a file with a large data size. Optionally, the data size of the first file is greater than or equal to a preset threshold. The first threshold is, for example, 1M, 4M, or 10M. In some cases, the device manages large files at the granularity of multiple data blocks to facilitate collaborative updates at the data block level. Small files, on the other hand, are relatively small and have high file read / write efficiency, so collaborative updates at the data block level are not necessary. Of course, this application also applies to the case where small files are managed at the granularity of multiple data blocks.

[0210] In some possible implementations, when the first device obtains first data from the second device, the first device may send or publish a data acquisition request, which requests the acquisition of the first data from the second device. The second device receives the data acquisition request from the first device and provides the first data to the first device.

[0211] Step S506: The first device updates the first file stored on the first device using the first data.

[0212] For example, referring to Figure 6, the first device obtains the data of block 2 from the second device and updates the local first file. Specifically, it updates the data in block 2 of the first file. Alternatively, the first device obtains first data from the second device and updates the data at the corresponding position in block 2 of the first file.

[0213] In the embodiment shown in Figure 6, the device can merge modifications to files at the data block level, support collaborative updates of file data in the file system by multiple devices, realize file synchronization between multiple data synchronization systems, solve the problem of extra storage space occupation caused by conflicting modifications to files, and improve the user experience.

[0214] In one possible implementation, the first device updates the last modification time of the target data block in the first extended metadata of the first file to the first modification time. In the above implementation, the first device updates the last modification time of the target data block to the latest first modification time, maintaining the accuracy and real-time performance of the extended metadata of the first file and improving system stability. Continuing with Figure 6 as an example, after the first device obtains the data of block 2 from the second device, it updates the local timestamp value of block 2 in the extended metadata of the first file to T1.

[0215] In one possible implementation, after updating the content of the data block, the first device also issues a modification confirmation to acknowledge that the modification update by the second device has been accepted. For example, the first device sends a first modification confirmation message, which instructs the first device to confirm the first modification.

[0216] Optionally, the first modification confirmation message includes one or more of the following: the identifier of the operating device for the first modification, the identifier of the first document on the operating device, the identifier of the device to which the first document belongs, the identifier of the first document on the device to which it belongs, and the first modification.

[0217] Please refer to Figure 8, which is a schematic diagram of the format of a modification confirmation message provided in an embodiment of this application. The modification confirmation message includes one or more of the following information: operation site identifier, operation site file identifier, offset, length, operation timestamp, data modification site identifier, and owner site identifier or owner site file identifier. Some of these fields can be referred to the relevant description in Figure 7 above. The modification confirmation message may optionally also include the identifier of the data modification site, which is used to indicate the identifier of the site that modified the data. For example, if the data modification is initiated by a second device, the site that modified the data is the identifier of the second device. Of course, in the modification confirmation message, the values ​​of some fields may be the same. For example, the values ​​of the operation site identifier and the data modification site identifier fields may be the same.

[0218] In some possible implementations, the modification confirmation message may also include a modification confirmation indication. The modification confirmation indication is used to instruct the first device to confirm the modification. For example, the modification confirmation message may include a first field, whereby the first device confirms the modification when the field takes a first value (e.g., 1).

[0219] In one possible implementation, if the first modification time is later than the target time, a first modification rejection message is sent. This first modification rejection message instructs the first device to reject the first modification. Exemplarily, the first modification rejection message includes a first modification location, a first modification time, and a modification rejection indication. This rejection indication indicates that the modification is rejected. For example, the modification rejection message includes a first field. When the first field takes a second value (e.g., 0), it indicates that the first device rejects the modification. Further, when the field takes a first value (e.g., 1), it indicates that the first device confirms the modification. Optionally, the first modification rejection message also includes one or more of the following: an identifier of the operating device, an identifier of the first file on the operating device, an identifier of the device to which the first file belongs, and an identifier of the first file on the device to which it belongs.

[0220] Furthermore, the second device receives a first modification rejection message from the first device, which instructs the first device to reject the first modification. In response to the first modification rejection message, the content of the second data block is deleted or the second data block is marked as old data.

[0221] In one possible implementation, the first device is the owner device of the first file. The owner device is responsible for maintaining the complete data content of the file, thus enabling the first device to update the data content of the first file based on modifications made by other devices. By defining the owner device of a file, a designated responsible device can be defined for a file in a system with multiple devices, improving data consistency in the multi-device system and enhancing the user experience.

[0222] The foregoing embodiments described the operations performed between the first and second devices to ensure file data consistency when the second device updates the data of the first file. However, the first device itself may also update the data of the first file. In one possible scenario, if the first device is the owner of the first file, and it updates the data of the first file, it also needs to publish an update message. However, this message only indicates that the data of the first file has been updated by other devices. Other devices can retrieve the updated data of the first file from the first device as needed, or they can choose not to perform the new data retrieval operation or other operations based on their own needs.

[0223] In one possible implementation, the first device receives modification operation information for a second modification of a first file, the modification operation information including second data. The first device updates the first file based on the second data, and determines a first data block among multiple data blocks based on the position of the second data in the first file. The first device updates the last modification time corresponding to the first data block in the extended metadata of the first file to a second modification time, the second modification time indicating the operation time of the second modification. In this implementation, when a user modifies a file on the first device, it can trigger an update of the file's data, and simultaneously update the local last modification time of the data block involved in the modification. This implementation can maintain the accuracy and real-time performance of the extended metadata of the first file, improving system stability.

[0224] Furthermore, the first device sends a second update message. The first update message indicates that the first device has performed a second modification on the first file. The second update message includes the identifier of the device performing the second modification, the identifier of the device that owns the first file, the location of the second modification, and the time of the second modification. The location of the second modification indicates the position of the second data in the first file. Both the identifier of the device performing the second modification and the identifier of the device that owns the first file are identifiers of the first device. The above embodiment describes a synchronization method when the owner device modifies the data in the file. The owner device can send an update message indicating the location of the updated data and the time of the update. Furthermore, other devices can discard local data blocks based on the location of the data update and the timestamp of the update.

[0225] In one possible implementation, since multiple devices may exist in the data synchronization system, and multiple devices may initiate modifications to the first file, taking the second and third devices as examples, when modifications involve the same data blocks, the first device makes a judgment based on the timestamp and retains the modification with the later modification time. Here, "same" does not necessarily mean completely identical; it may also mean partial overlap.

[0226] As a possible example, the first device receives a third update message from the third device. This message indicates that a first file stored on the third device has undergone a third modification. The third update message includes the third modification location and the third modification time. The third modification location indicates the position of the third data corresponding to the third modification in the first file, and the third modification time indicates the operation time of the third modification. Based on the third modification location, the first device determines the data block corresponding to the third data among multiple data blocks. If the first device determines that the data block corresponding to the third data conflicts with the target data block, and the first modification time is later than the third modification time, it retrieves the first data from the second device. For example, if the operation time of the third modification is T2, and the third modification also involves data block 2, and T2 is less than T1, then the first device does not retrieve the data for block 2 from the third device.

[0227] It should be noted that the actions performed by the first, second, and third devices described above are not entirely fixed, but are illustrative examples based on the modification process of the first file. In some cases, when modifying other files, the first device may also perform the operations performed by the second device. For example, if the second file belongs to the second device, and the second file stored on the first device is modified, the first device sends an update message to the second device, and the second device receives the update message and performs the corresponding processing (refer to the actions performed by the first device in Figure 5).

[0228] In some possible implementations, other devices can receive a first modification confirmation message sent by the first device. Further, for the operating device (i.e., the device that initiated the modification), such as the second device, a read request can be created to read the latest data block content. Optionally, for a non-operating device, such as a third device, the timestamp in the modification confirmation message can be compared with the timestamp of its own modification of the data block. If its modification conflicts with the modifications of other devices, and the timestamp of the modification confirmation message is later than its own timestamp of modification, the non-operating device can invalidate its own modification of the data block, i.e., the locally modified data block is discarded. For example, the modified data block can be deleted, the modification rolled back, or the data marked as discarded. In some schemes, for a non-operating device, when there is a read request for the first file, it can obtain the latest data block again from the first device.

[0229] In one possible implementation, for the device that sent the update message, such as the second device, a first tag message can be generated for the first data (or target data block). The first tag message is used to mark that the first data will not be evicted before the first device obtains the first data from the second device. For example, the second device generates a write message (considered the first tag message) for the target data block (such as block 2), which can protect the target data block from being evicted.

[0230] In one scenario, after the first device accepts the modification from the second device, the first device no longer needs to protect the target data block from being obsolete; in this case, the first tag message can be deleted. That is, after receiving the first modification confirmation message from the first device, the second device deletes the first tag message. The first modification confirmation message is used to instruct the first device to confirm the first modification. In some cases, after receiving the first modification confirmation message from the first device, the second device also generates read information for the first data or the data block containing the first data.

[0231] In some cases, even if the first device has not received the first modification, such as when it has received a later modification targeting the target data block, the second device may also delete the aforementioned first marker message. For example, the second device receives a first message from the first device indicating that the first file has been modified a fourth time. This first message includes the location and time of the fourth modification; the location indicates the position of the fourth data corresponding to the fourth modification in the first file, and the time indicates the operation time of the fourth modification. The second device identifies a second data block among multiple data blocks, which corresponds to the fourth data. If it determines that the second data block conflicts with the target data block, and the fourth modification time is later than the first modification time, the first marker message is deleted.

[0232] Taking the third device as an example, the third device sends a third update message, indicating that it has performed a third modification on the first file. This third modification also involves the target data block, but the operation time of the third modification is less than the operation time of the first modification. At this time, the third device also creates a first marker for the target data block. After receiving the first modification confirmation message, the third device compares the operation time of the first modification indicated in the confirmation message with the operation time of the third modification, and can determine that the third modification has not been received. At this point, the first marker is deleted.

[0233] In some possible implementations, the devices storing the file data all maintain extended metadata for the first file. Taking a second device as an example, the second device stores some or all of the data of the first file, and maintains second extended metadata for the first file. The second device can receive a first message from the device to which the first file belongs. When the first message indicates that the first file has been modified, and the modification operation time is later than the local last modification time of the data block involved in the modification, the second device deletes the content of the data block involved in the modification or marks the locally stored data block involved in the modification as old data.

[0234] As one possible implementation example, the second device receives a first message from the first device. This first message indicates that the first file has been modified. The first message includes the modification location and modification time. The modification location indicates the position of the data corresponding to the modification within the first file, and the modification time indicates the time of the modification operation. The second device identifies a second data block among multiple data blocks. This second data block corresponds to the data whose modification occurred. If the modification time is later than the first time, the second device deletes the content of the second data block or marks it as old data. The first time is the last modification time corresponding to the second data block. Optionally, the first message can be an update message initiated by the first device, or it can be a modification confirmation message initiated by the first device. The content of the update message and the modification confirmation message are described above, and will not be illustrated here.

[0235] The embodiments shown in Figure 6 above describe various possible implementations. These possible implementations can be combined, and one possible combination implementation is described below with reference to Figure 9. The data synchronization system shown in Figure 9 includes a data cache site 1, a data cache site 2, and a data owner site, which are given from the perspective of File1. That is, data cache site 1, data cache site 2, and data owner site all store the data of File1. Furthermore, all three maintain extended metadata of the first file for ease of description. The following description is based on the following: the data owner site maintains the first extended metadata, data cache site 1 maintains the second extended metadata, and data cache site 2 maintains the third extended metadata. All of the above extended metadata are the local last modification time of the data blocks of the first file recorded locally on the device.

[0236] File1 stored on data cache site 1 undergoes a first modification. The data corresponding to the first modification is "first data," the operation time of the first modification is T1, and the first modification involves a target data block, such as block 2. File1 stored on data cache site 2 undergoes a third modification. The data corresponding to the third modification is "third data," the operation time of the third modification is T2, and the third modification involves a target data block, such as block 2. Optionally, data cache site 1 can update the last modification time of the target data block in the second extended metadata to T1. Similarly, data cache site 2 can update the last modification time of the target data block in the third extended metadata to T2.

[0237] Data caching station 1 sends a first update message to the message relay station (as shown in step ①). The first update message includes the data location of the first data and the first modification timestamp (indicating T1). Similarly, data caching station 2 sends a third update message to the message relay station (as shown in step ②). The third update message includes the data location of the third data and the third modification timestamp (indicating T2). Please refer to Figure 7 for the format of the modification request message.

[0238] The data ownership station receives the first update message and the third update message from the message relay station (as shown in step ③). The data ownership station compares the modification timestamp in the update message with the local timestamp of the target data block in the extended metadata of the first file. If the timestamp in the update message is greater than the local timestamp of the target data, the data ownership station retrieves the corresponding data block from the data cache station.

[0239] For example, T1 is greater than the local timestamp of the target data block, while T2 is less than the local timestamp of the target data block. Therefore, the data ownership site needs to retain the modifications made by the data cache site 1 at time T1. Specifically, the data ownership site reads the corresponding data block from the data cache site 1 and writes it to the local data block according to the data position of the first data in the first update message (as shown in step ④, the pull update data step), for example, obtaining the data of block 2 and writing it locally.

[0240] Furthermore, the data ownership station sends a modification confirmation message for File1 to the message relay station (as shown in step ⑤), indicating confirmation of the first modification on the second device. The modification confirmation message includes the location of the first modification and the operation time of the first modification. Other data caching stations can receive the modification confirmation message (as shown in step ⑥).

[0241] Optionally, after receiving a modification confirmation message, if the modification corresponding to the confirmation message was initiated locally, the data cache site marks the corresponding local data block as read information. This read information can be subsequently discarded by the system. For example, for data cache site 1, after receiving the modification confirmation message, the data cache site knows that the first modification has been received, so data cache site 1 can mark its local block 2 as read information, which can be subsequently discarded.

[0242] If the modification confirmation message corresponds to a modification initiated by another site, the device compares the timestamp in the modification confirmation message with the timestamp corresponding to the locally modified data block (e.g., T2). If the timestamp in the modification confirmation message is greater than or equal to the timestamp corresponding to the locally modified data block, the locally modified data block is discarded. For example, for data cache site 2, after receiving the modification confirmation message, since the modification confirmation message includes the operation location and operation time of the first modification, data cache site 2 can determine the target data block involved in the first modification based on the operation location of the first modification, and thus can look up the third extended metadata to obtain the last modification time of the target data block recorded on data cache site 2. Since the operation time of the first modification is later than the last modification time of the target data block recorded in the third extended metadata, data cache site 2 can know that its third modification has not been confirmed by the first device. At this time, data cache site 2 can delete the content of the target data block or mark the target data block as old data. Furthermore, when a user of data cache site 2 subsequently initiates a read request for the first file, data cache site 2 can retrieve the data of the first file again from the data home site, or retrieve the data of the data block marked as old data.

[0243] To implement the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0244] It is understood that the multiple devices provided in the embodiments of this application, such as data synchronization devices, are for implementing the functions in the above method embodiments, and include hardware structures, software modules, or combinations of hardware structures and software structures to perform each function.

[0245] Those skilled in the art will readily recognize that the units and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the foregoing method embodiments using different device implementations in different application scenarios, and such different device implementations should not be considered beyond the scope of the embodiments of this application.

[0246] The embodiments of this application can divide the device into functional modules. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in the embodiments of this application is illustrative and is only a logical functional division; in actual implementation, there may be other division methods.

[0247] Please refer to Figure 10, which is a schematic diagram of a data synchronization device provided in an embodiment of this application. The data synchronization device 100 includes a communication module 1001 and a processing module 1002. The communication module 1001 is used to receive and / or send messages, and also to perform other operations requiring communication with external systems. The processing module is used to process messages. The processing module 1002 is used to process information, such as performing one or more operations like comparison, decision-making, updating, generating, determining, writing, or reading.

[0248] The data synchronization device 100 is used to implement the aforementioned data synchronization method, such as the data synchronization method shown in FIG5 or FIG9. Optionally, the data synchronization device 100 can be a standalone device, such as a computing device, or a module in a standalone device. For example, the data synchronization device 100 can be a first data synchronization device in the first device shown in FIG1, or a second data synchronization device, or a third data synchronization device.

[0249] In one possible design, the data synchronization device 100 is used to implement the operations performed by the first device or data home station in the embodiment shown in FIG5 or FIG9.

[0250] In one possible implementation, the communication module 1001 is used to acquire a first update message from the second device. The processing module 1002 is used to determine a target data block among a plurality of data blocks, and if the first modification time is later than the last modification time corresponding to the target data block, acquire first data from the second device and use the first data to update the first file stored on the first device. Here, the last modification time corresponding to the target data block is the last modification time corresponding to the target data block. Optionally, the process of acquiring the first data may also utilize the communication module 1001. The specific functions performed by the above modules can be found in the description in the foregoing method embodiments.

[0251] As another possible implementation, the processing module 1002 is further configured to update the last modification time corresponding to the target data block in the first extended metadata of the first file to the first modification time.

[0252] As another possible implementation, the communication module 1001 is also used to send a first modification confirmation message, which is used to instruct the first device to confirm the first modification.

[0253] As another possible implementation, the first modification location includes an offset and a data length. The processing module 1002 is also used to determine a target data block based on the data location range corresponding to multiple data blocks and the data location range indicated by the offset and data length.

[0254] As another possible implementation, the processing module 1002 is also configured to obtain a target data block from the second device, wherein the data in the target data block includes the first data.

[0255] As another possible implementation, the processing module 1002 is also used to obtain first data from the second device based on a first modified position, the first modified position including an offset and a data length.

[0256] As another possible implementation, the communication module 1001 is further configured to receive modification operation information for a second modification of the first file, the modification operation information including second data. The processing module 1002 is further configured to: update the first file based on the second data, determine a first data block among a plurality of data blocks based on the position of the second data in the first file, and update the last modification time corresponding to the first data block in the first extended metadata to a second modification time, the second modification time being used to indicate the operation time of the second modification.

[0257] As another possible implementation, the communication module 1001 is also used to send a second update message, which indicates that the first file stored on the first device has been modified in a second way.

[0258] As another possible implementation, the communication module 1001 is further configured to acquire a third update message from a third device. The processing module 1002 is further configured to: determine the data block corresponding to the third data among multiple data blocks based on the third modification position; and, if it is determined that the data block corresponding to the third data conflicts with the target data block and the first modification time is later than the third modification time, acquire the first data from the second device.

[0259] For a detailed description of the operation in the above embodiments, please refer to the preceding descriptions, such as the descriptions of the method embodiments and system architecture sections.

[0260] In another possible design, the data synchronization device is used to implement the operations performed by the second device or data cache site 1 in the embodiment shown in Figure 5 or Figure 9.

[0261] In one possible implementation, the communication module 1001 is used to acquire modification operation information of a first modification to the first file, the modification operation information of the first modification including first data. The communication module 1001 is also used to send a first update message so that the first device can acquire the first data from the second device.

[0262] As another possible implementation, the data synchronization device further includes a processing module 1002. The processing module 1002 is used to update the last modification time of the target data block recorded in the second extended metadata to the first modification time, wherein the target data block is the data block corresponding to the first data among a plurality of data blocks.

[0263] As another possible implementation, the communication module 1001 is further configured to receive a first modification confirmation message from the first device after the first device obtains the first data from the second device, the first modification confirmation message being used to instruct the first device to confirm the first modification.

[0264] As another possible implementation, the communication module 1001 is further configured to receive a first modification confirmation message from the first device when the first modification time is later than the last modification time corresponding to the target data block. The first modification confirmation message is used to instruct the first device to confirm the first modification.

[0265] As another possible implementation, the communication module 1001 is also configured to receive a data acquisition request from the first device, the data acquisition request being used to request the acquisition of first data on the second device in order to update the first file on the first device.

[0266] As another possible implementation, the communication module 1001 is further configured to acquire a first message from the first device, the first message indicating that the first file has been modified a fourth time. The processing module 1002 is further configured to: determine a second data block among a plurality of data blocks, the second data block being the data block corresponding to the fourth modification. If it is determined that the second data block conflicts with the target data block, and the operation time of the fourth modification is later than the first modification time, the second data block is deleted or marked as old data.

[0267] As another possible implementation, the data synchronization device further includes a processing module 1002. The processing module 1002 is used to generate a first tag message for the first data, the first tag message being used to mark that the first data will not be discarded before the first device obtains the first data from the second device.

[0268] As another possible implementation, the data synchronization device further includes a processing module 1002. The processing module 1002 is used to delete a first flag message after receiving a first modification confirmation message or a first message from the first device. The first modification confirmation message is used to instruct the first device to confirm the first modification, and the first message is used to instruct the first device to confirm a modification whose operation time is later than the operation time of the first modification.

[0269] For a detailed description of the operation in the above embodiments, please refer to the preceding descriptions, such as the descriptions of the method embodiments and system architecture sections.

[0270] Figure 11 shows a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device 110 is a device with computing capabilities. The device here can be a physical device, such as a controller, processor, server (such as a rack server), host, etc., or it can be a virtual device, such as a virtual machine, container, etc.

[0271] As shown in Figure 11, the computing device 110 includes a processor 1102 and a memory 1101, and optionally includes a connection line 1104 and a communication interface 1103. The processor 1102 and the memory 1101 communicate with each other via the connection line 1104. It should be understood that this application does not limit the number of processors and memories in the computing device 110.

[0272] Memory 1101 provides storage space, which may optionally store application data, user data, operating system, and computer programs. Memory 1101 may include volatile memory, such as random access memory (RAM). Memory 1101 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0273] Processor 1102 is a module for performing calculations and may include any one or more of the following: controller (e.g., memory controller), central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), coprocessor (to assist the central processing unit in completing corresponding processing and applications), application-specific integrated circuit (ASIC), microcontroller unit (MCU), virtual machine, container, etc.

[0274] Communication interface 1103 is used to provide information input or output to at least one processor. And / or, communication interface 1103 can be used to receive data transmitted externally and / or transmit data externally. Communication interface 1103 can be a wired link interface, including an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, and other wireless communication technologies, etc.). Optionally, communication interface 1103 may also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0275] Connection line 1104 is used to connect various modules in the device to enable data transmission between different modules. Exemplarily, connection line 1104 includes a bus, such as a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one connection line or one type of connection line. Connection line 1104 may include a path for transmitting information between various components of computing device 110 (e.g., memory 1101, processor 1102, communication interface 1103).

[0276] In this embodiment, memory 1101 stores executable instructions, and processor 1102 executes these executable instructions to implement the aforementioned data synchronization method, such as the method shown in one or more embodiments of FIG. 5 or FIG. 9 and any optional implementation thereof, for example, executing a method executed by a first device, and / or executing a method executed by a second device, and / or executing a method executed by a third device. That is, memory 1101 stores instructions for executing the data synchronization method.

[0277] This application embodiment also provides a computing device cluster, which includes at least one computing device 110, each computing device 110 including a processor 1102 and a memory 1101. The processor 1102 of the at least one computing device 110 is used to execute instructions stored in the memory 1101 of the at least one computing device 110 to cause the computing device cluster to implement the aforementioned data synchronization method, such as the method shown in one or more embodiments of FIG. 5 or FIG. 9 and any optional implementation thereof, for example, executing the method executed by a first device, and / or executing the method executed by a second device, and / or executing the method executed by a third device.

[0278] This application also provides a storage device, which includes a storage unit and a data synchronization device as shown in FIG10 or a computing device as shown in FIG11. The storage unit is used to provide storage space, and the data synchronization device or computing device is used to implement the aforementioned storage management method, such as the method shown in one or more embodiments of FIG5 or FIG9 and any optional implementation thereof, for example, executing the method executed by the first device.

[0279] In some scenarios, the aforementioned storage devices can be storage products provided by storage vendors. For example, storage devices may include storage products from Dorado or Pacific, etc.

[0280] This application also provides a computer-readable storage medium storing instructions that, when executed on at least one processor (or computing device), implement the aforementioned storage management method, such as the method shown in one or more embodiments of FIG5 or FIG9 and any of their optional implementations, for example, executing a method performed by a first device.

[0281] This application also provides a computer program product, which includes computer instructions for implementing the aforementioned storage management method, such as the method shown in one or more embodiments of FIG5 or FIG9 and any optional implementation thereof, for example, executing a method performed by a first device.

[0282] In addition, a few additional points need to be made regarding this application:

[0283] I. The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

[0284] 2. Unless otherwise stated, “multiple” means two or more.

[0285] 3. Unless otherwise specified or in case of logical conflict, the terms and / or descriptions in different embodiments of this application are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0286] IV. The various numerical designations used in this application are merely for descriptive convenience and are not intended to limit the scope of protection of this application. The magnitude of the serial numbers used in this application does not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic. For example, the terms "first," "second," "third," "fourth," and other various terminology (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0287] Furthermore, any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0288] V. The terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are expressly listed, but may include other steps or modules that are not expressly listed or that are inherent to such process, method, product or device.

[0289] VI. In this application, "for indicating" can be understood as "enabling". "Enabling" can include direct enabling and indirect enabling. When describing information for enabling A, it can include whether the information directly enables A or indirectly enables A, but does not necessarily mean that the information carries A.

[0290] The information that enables the information is called the information to be enabled. In the specific implementation process, there are many ways to enable the information to be enabled, such as, but not limited to, directly enabling the information to be enabled, such as the information to be enabled itself or its index. It can also be indirectly enabled by enabling other information, where there is a relationship between the other information and the information to be enabled. It can also enable only a part of the information to be enabled, while the other parts are known or pre-agreed upon. For example, enabling specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing enabling overhead to some extent. Simultaneously, common parts of various pieces of information can be identified and enabled uniformly to reduce the enabling overhead caused by individually enabling the same information.

[0291] VII. In this application, "predefined" may include preconfiguration. For example, predefining certain information means that the information is calculated or received in advance before performing an action that uses the information. The "predefined" can be implemented by pre-storing corresponding codes, tables, or other means that can be used to indicate relevant information in the device (e.g., controller or vehicle). This application does not limit the specific implementation method.

[0292] 8. The term "storage" or "preservation" in this application can refer to storage in one or more memory devices. These memory devices can be separately configured or integrated into an encoder or decoder, processor, or communication device. Alternatively, some memory devices can be separately configured, while others can be integrated into a decoder, processor, or communication device. The type of memory can be any form of storage medium, and this is not limited.

[0293] 9. The arrows or boxes indicated by dashed lines in the schematic diagrams in the accompanying drawings of this application represent optional steps or optional modules.

[0294] 10. Unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. In this application, "and / or" is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

Claims

1. A data synchronization method, characterized in that, The first device is used to manage the reading and writing of the first file on multiple devices at the granularity of data blocks based on the first extended metadata of the first file. The first extended metadata includes the last modification time corresponding to multiple data blocks of the first file. The multiple devices include the first device and the second device. The method includes: Obtain a first update message from the second device. The first update message is used to indicate that the first file stored on the second device has been modified. The first update message includes a first modification location and a first modification time. The first modification location is used to indicate the location of the first data corresponding to the first modification in the first file. The first modification time is used to indicate the operation time of the first modification. Determine a target data block among the plurality of data blocks, wherein the target data block is the data block corresponding to the first data; If the first modification time is later than the last modification time corresponding to the target data block, the first data is obtained from the second device; The first file stored on the first device is updated using the first data.

2. The method according to claim 1, wherein The method further includes: Update the last modification time of the target data block in the first extended metadata to the first modification time.

3. The method according to claim 1 or 2, characterized in that, The method further includes: A first modification confirmation message is sent, which instructs the first device to confirm the first modification.

4. The method according to any one of claims 1 to 3, characterized in that The first modified position includes an offset and a data length. The offset is used to indicate the offset of the starting position of the first data relative to the starting position of the first file, and the data length is used to indicate the length of the first data. Determining the target data block among the plurality of data blocks includes: The target data block is determined based on the data location range corresponding to the plurality of data blocks and the data location range indicated by the offset and the data length.

5. The method according to any one of claims 1 to 3, characterized in that, The first modification location includes indication information of the target data block.

6. The method according to any one of claims 1-5, characterized in that, Obtaining the first data from the second device includes: The target data block is obtained from the second device, and the data in the target data block includes the first data; or, The first data is obtained from the second device based on the first modified position, wherein the first modified position includes an offset and a data length.

7. The method according to any one of claims 1 to 6, characterized in that, The first device is the owner device of the first file, the second device is the cache device of the first file, and the first device is used to manage the complete data content of the first file.

8. The method according to claim 7, characterized in that The method further includes: Receive modification operation information for a second modification of the first file, wherein the modification operation information for the second modification includes second data; The first file stored on the first device is updated based on the second data; The first data block among the plurality of data blocks is determined based on the position of the second data in the first file; The last modification time corresponding to the first data block in the first extended metadata is updated to the second modification time, which is used to indicate the operation time of the second modification.

9. The method according to any one of claims 1 to 8, wherein The method further includes: Send a second update message, which is used to indicate that the first file stored on the first device has been modified. The second update message includes the identifier of the device that performed the second modification, the identifier of the device to which the first file belongs, the second modification location and the second modification time. The second modification location is used to indicate the location of the second data in the first file, and the second modification time is used to indicate the operation time of the second modification. The identifier of the device performing the second modification and the identifier of the device to which the first file belongs are both the identifier of the first device.

10. The method according to any one of claims 1-9, characterized in that The plurality of devices also includes a third device. The method further includes: Obtain a third update message from the third device. The third update message is used to indicate that the first file stored on the third device has been modified a third time. The third update message includes the third modification location and the third modification time. The third modification location indicates the location of the third data corresponding to the third modification in the first file. The third modification time is used to indicate the operation time of the third modification. Obtaining the first data from the second device includes: Based on the third modification position, determine the data block corresponding to the third data among the plurality of data blocks; If it is determined that the data block corresponding to the third data conflicts with the target data block, and the first modification time is later than the third modification time, then the first data is obtained from the second device.

11. The method according to any one of claims 1-10, characterized in that, The first device and the second device transmit messages through a message relay station.

12. The method according to any one of claims 1 to 11, characterized in that, The last modified time includes a timestamp or a version number, which changes monotonically over time.

13. A data synchronization method, characterized in that, The method is applied to a second device, which maintains a second extended metadata of the first file. The second extended metadata includes the last modification time corresponding to multiple data blocks of the first file recorded by the second device. The method includes: Obtain modification operation information of the first modification to the first file, wherein the modification operation information of the first modification includes first data; Send a first update message to enable the first device to obtain the first data from the second device; The last modification time of the target data block recorded in the second extended metadata is updated to the first modification time, and the target data block is the data block that corresponds to the first data among the plurality of data blocks; The first update message is used to indicate that the first file stored on the second device has been modified. The first update message includes a first modification location and a first modification time. The first modification location indicates the location of the first data corresponding to the first modification in the first file, and the first modification time indicates the operation time of the first modification.

14. The method according to claim 13, characterized in that, The first device maintains first extended metadata for the first file, the first extended metadata including the last modification time corresponding to each of the plurality of data blocks recorded by the first device, and the method further includes: If the first modification time is later than the last modification time corresponding to the target data block, a first modification confirmation message is received from the first device. The first modification confirmation message is used to instruct the first device to confirm the first modification. The last modification time corresponding to the target data block is the last modification time of the target data block recorded in the second extended metadata.

15. The method according to claim 13, characterized in that, The method further includes: Obtain a first message from the first device, the first message being used to indicate that the first file stored on the first device has been modified a fourth time, the first message including the fourth modification location and the fourth modification time, the fourth modification location indicating the location of the fourth data corresponding to the fourth modification in the first file, and the fourth modification time indicating the operation time of the fourth modification; A second data block is determined from the plurality of data blocks, wherein the second data block is the data block corresponding to the fourth data; If it is determined that the second data block conflicts with the target data block, and the fourth modification time is later than the first modification time, the content of the second data block is deleted or the second data block is marked as old data; the first modification time is the last modification time of the second data block recorded in the second extended metadata.

16. The method according to any one of claims 13-15, characterized in that, Before receiving the first modification confirmation message from the first device, the method further includes: A first tag message is generated for the first data, the first tag message being used to mark that the first data will not be discarded before the first device obtains the first data from the second device.

17. The method according to claim 16, characterized in that, The method further includes: After receiving a first modification confirmation message or a first message from the first device, the first flag message is deleted. The first modification confirmation message is used to instruct the first device to confirm the first modification, and the first message is used to instruct the first device to confirm a modification whose operation time is later than the operation time of the first modification.

18. The method according to claim 15, characterized in that, The method further includes: In response to a data read request, target data is obtained from the first device, and the last modification time of the second data block in the second extended metadata is updated. The data read request is used to indicate that the target data is read from the second device, and the target data corresponds to the second data block.

19. The method according to any one of claims 13-18, characterized in that, The first device is the owner device of the first file, the second device is the cache device of the first file, and the first device is used to manage the complete data content of the first file.

20. A data synchronization device, characterized in that, The data synchronization device includes a communication module and a processing module, and is used to implement the method according to any one of claims 1-12, or to implement the method according to any one of claims 13-19.

21. A storage device, characterized in that, The storage device includes a processor and a memory, and the storage device further includes a first storage unit or is connected to the first storage unit; The memory stores computer instructions, and the processor is used to invoke the computer instructions stored in the memory to implement the method according to any one of claims 1-12, or to implement the method according to any one of claims 13-19.

22. A data synchronization system, characterized in that, The data synchronization system includes a first device and a second device. The first device is used to implement the method according to any one of claims 1-12. The second device is used to implement the method according to any one of claims 13-19.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1-19.

24. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1-19.

Images