The invention discloses a multi-level batch text parallel deduplication method,
system and device and a storage medium, and the method comprises the steps: carrying out the preprocessing of text data, and carrying out the similarity text representation and content deduplication of sensitive hashing at a chapter level, a
paragraph level and a statement level; adding a priority
score and a sorting rule in a duplicate removal process to eliminate duplicate removal randomness; during content deduplication, a graph structure is created according to similarity text representation of sensitive Hash, a plurality of connected components are established on the graph structure by utilizing combination and separation of union sets, and a
target text is selected and reserved in each
connected component by adopting a dynamic Hash threshold value; in the process, data frame objects are respectively constructed for chapter-level,
paragraph-level and statement-level texts, unique identifiers are allocated, and the
time complexity and the required memory space are optimized by adopting a parallel deduplication strategy of block
delay calculation and pointer storage. According to the method, the duplicate removal speed and accuracy can be improved, and meanwhile, the problem of memory shortage faced by
mass duplicate removal is solved.