The application provides a log collection method and
system based on
parallel sorting, which specifically comprises the following steps: (1) data sampling: collecting
log data from different log sources, transmitting the
log data to a
data processing node, and using a
big data-oriented scalable regular sampling method to obtain representative samples from the
log data; (2)
parallel sorting: distributing the representative samples to multiple nodes in the
system according to certain rules, and sorting the log data on each node using a
parallel sorting algorithm; (3) parallel writing: collecting the sorted log data representative samples into a
file system in parallel; (4) monitoring and optimization: monitoring the space utilization of the
file system in real time through a
verification file and a checking tool, and regularly checking the integrity of the log data. The application uses a
big data-oriented scalable regular sampling method to preprocess and sample the log, and uses parallel sorting technology to efficiently collect the log data. Then, the sampled log sample data is distributed to multiple
processing nodes to realize parallel sorting operation. The application disperses sample storage and
processing tasks to each process for sorting and iteratively updates the
pivot element, thereby improving parallel efficiency, reducing
processing bottlenecks caused by uneven data distribution, and improving the overall performance and reliability of the
log management system.