Distributed Database Column Compression for Lower Node Transmission Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data transmission between nodes in a distributed database system results in high overheads and network load due to the transmission of uncompressed data.
Innovation Solution
A data node determines a compression algorithm based on the data type and distribution rule of a column's parameters, compresses the data using that algorithm, and sends compressed data to a target node, reducing transmission overheads and network load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If uncompressed data is transmitted between nodes, then data integrity is maintained, but transmission overheads are high and network load is heavy
Solution Approach 1:
The patent applies compression algorithms to change the parameter of data size before transmission. By selecting appropriate compression algorithms based on data types and distribution rules, the system reduces the quantity of data (parameter 26) while maintaining integrity, thereby reducing transmission overheads (parameter 22) and network load
Solution Approach 2:
The patent performs data compression in advance before transmission occurs. The coordinator node determines compression algorithms based on data types and distribution rules, pre-compresses the data, and then transmits the compressed form. This preliminary compression action reduces the data size that needs to be transmitted, addressing the contradiction between maintaining data integrity and reducing transmission overhead
2Productivity
If compression algorithms are applied to all columns, then data transmission efficiency is improved, but system complexity increases
Solution Approach 1:
The patent applies compression algorithms selectively to specific columns based on their data types and distribution rules, rather than uniformly to all columns. The coordinator node analyzes each column's characteristics and determines the most suitable compression algorithm only where applicable. This localized approach improves data transmission efficiency for suitable columns while avoiding unnecessary complexity in columns that don't benefit from compression
Solution Approach 2:
The system changes the parameter of algorithm selection based on column-specific characteristics. By determining compression algorithms dynamically based on data types and distribution rules of each column, the system achieves higher transmission efficiency where applicable while maintaining simplicity where compression is not beneficial, thus resolving the contradiction between productivity improvement and system complexity
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Embodiments of the present invention relate to the database field, and in particular, to a data transmission method and apparatus, so as to reduce overheads for data transmission between nodes in a distributed database and lighten network load. In the embodiments of the present invention, a DN determines one to-be-transmitted column that has a distribution rule, and the distribution rule of parameters in the to-be-transmitted column; the DN determines, according to a data type of the parameters of the to-be-transmitted column and the distribution rule of the parameters in the to-be-transmitted column, a compression algorithm corresponding to the to-be-transmitted column; the DN compresses the to-be-transmitted column by using the compression algorithm; and the DN sends a compressed to-be-transmitted column to a target node. In this way, the overheads for the data transmission between the nodes are reduced, and the network load is lightened.