The invention discloses a large-scale
data set increment synchronization method based on
cloud storage. The method comprises the following steps: designing a cloud
object storage intermediary transmission architecture, designing an offset mapping
file system realized based on a user mode
file system FUSE, and designing a sampling type file digest
algorithm; comprises: generating a
data description file; establishing connection with
cloud storage, namely a transit hub; generating a transmission
task list; performing
task management and state maintenance; mounting the offset mapping
file system, mapping the file into an independent virtual file in the
virtual file system from a specified position, and reading and transmitting the virtual file; executing
data transmission; and carrying out increment merging and
data verification. The invention provides an efficient, low-cost and reliable large-scale
data set cross-ocean synchronization solution, and can solve the problems of high flow cost, limited cross-border bandwidth, unstable
network delay and the like in cross-ocean
data synchronization in an international network environment.