Bulk Insert Optimization in Distributed Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed database systems experience increased cross-node communication and load on processing nodes during bulk data insertions, particularly when handling large volumes of data, due to multiple hops of data transfer between index servers.
Innovation Solution
A system that identifies target index servers based on partition criteria, partitions data records into subsets, and inserts these subsets in parallel directly to the respective target index servers, reducing network load and congestion by processing data locally.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data records are pruned at the index server selected by the client and uploaded through multiple hops, then data can be inserted into partitioned tables, but cross-node communication increases and load on processing nodes increases
Solution Approach 1:
The client device performs preliminary actions by identifying target index servers and partitioning data records before upload, based on partition criteria retrieved in advance. This pre-processing eliminates the need for intermediate routing hops during the actual insert operation, as data is directly uploaded to the correct destination servers from the start.
Solution Approach 2:
The system segments the bulk insert process into distinct phases: retrieving partition criteria, identifying target servers, partitioning data records into subsets, and uploading to respective servers. This segmentation allows parallel processing of multiple data subsets simultaneously, improving overall productivity while reducing network load through direct connections.
2Adaptability or versatility
If data records are transferred through multiple hops between index servers, then distributed database insertion can be achieved, but network congestion increases
Solution Approach 1:
The system performs preliminary identification of target index servers and partitioning of data records before the actual upload process. By determining the destination servers in advance based on partition criteria, the system eliminates intermediate routing hops, enabling direct uploads that maintain distributed database adaptability while significantly improving insertion speed.
Solution Approach 2:
The invention changes the network communication dimension from multi-hop sequential transfer to direct parallel connections. Multiple data subsets are uploaded simultaneously to different target index servers through direct connections, transforming the network topology utilization from hierarchical routing to parallel distributed access.
3Quantity of substance
If conventional bulk insert methods are used with multiple hops, then data can be distributed across index servers, but load balancing deteriorates
Solution Approach 1:
The system segments the data records into multiple subsets based on partition criteria, with each subset destined for a specific target index server. This segmentation enables balanced distribution of data across multiple servers, as each server receives a proportional share of the total data load, improving load balancing while maintaining effective data distribution.
Solution Approach 2:
The client device performs preliminary partitioning of data records into subsets corresponding to different target index servers before upload. This pre-partitioning ensures that data is evenly distributed across the distributed database infrastructure from the outset, achieving both data distribution and load balance simultaneously.
Data Source
AI summary
Described herein is a system and method for initiating a bulk insert in a distributed database. The system described herein provides for a client to identify target index servers of the data records by examining the data records. The system identifies a partition criterion of the target index servers and partitions the data records based on a partition criterion, into subsets of data records. The system builds buckets including the subsets of data records that correspond to a partition. The system inserts each bucket including the subset of the data corresponding to a specific partition to the respective target index server where the specific partition is physically located and inserts the subset of data records in the table locally. The system inserts each bucket in each respective target index server in parallel.