The invention relates to the technical field of
data processing, and particularly discloses a discrete data collection method and
system based on hierarchical aggregation, by constructing a hierarchical aggregation architecture of computing nodes, intermediate aggregation nodes and a
central node, the computing nodes cache discrete data to a local pre-write log
queue, and the
central node stores the discrete data in the local pre-write log
queue; when a time window or a data volume threshold value is met, the data are subjected to
batch processing compression and then sent to the corresponding intermediate aggregation nodes, the intermediate aggregation nodes decompress the data, aggregate and deduplicate the data and send the aggregated data to the center node through a pre-write log and a
check point persistence state, and the center node can achieve non-repeated write-in of the aggregated data; and a
back pressure signal can be sent to adjust the transmission
rhythm of the intermediate aggregation node, and data collection is ensured to meet accurate primary
semantics through cooperation of a transaction ID, a
check point and an idempotent interface. The method can eliminate the I / O
bottleneck of the
central node, improves the collection efficiency and
data reliability, enhances the fault-tolerant capability, and is suitable for discrete data collection of a large-scale distributed
system.