The application discloses a
data security union
processing method and
system, belongs to the technical field of
data management, solves the problem that due to the inherent defect of hash
algorithm, hash collision cannot be completely avoided, the introduction of cross-
shard retrieval will destroy the
shard parallelism, significantly increase the merging complexity, and lead to the problem that the
shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed. Including the following steps: dynamic sharding, according to the data heat of the
primary key. Through the dynamic sharding, distributed
bloom filter conflict pre-checking and local redirection query technology, combined with multi-thread
parallel processing and real-time load balancing optimization, the proportion of cross-shard retrieval caused by hash collision is limited, the hash
collision rate is reduced while the shard parallelism is ensured, the super large
data set is merged, and the problem that the shard number matching may cause the actual associated data to be scattered in different shards due to hash collision and cross-shard retrieval is needed.