The invention relates to a
time series data acquisition method and device based on a dynamic time window and
fingerprint deduplication. The method comprises the following steps: establishing a
sliding time window with a
variable length by taking the current
system time as a reference, and only querying incremental
time sequence data in the window; adaptively zooming the window by calculating the ratio of the collected data volume to the expected volume in real time; setting the trigger interval as half of the length of the
sliding time window to form a time overlap, thereby capturing out-of-order data; key features of each piece of
time series data are extracted, and data fingerprints are generated; a
bloom filter is used for
primary screening, and accurate duplicate removal is carried out in a key-value storage
system; compared with an existing fixed
polling scheme, the method has the advantages that resources can be dynamically adjusted along with data flow rate,
system load is reduced, real-time performance is improved,
data integrity is guaranteed by overlapped windows, zero repeated acquisition is guaranteed by
fingerprint two-stage duplicate removal, and the method is applicable to efficient incremental synchronization scenes of transactional log-free
time sequence databases such as InfluxDB and the like.