Columnar Database Write Optimization via Disk Cylinder Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Columnar databases experience slow write speeds due to the need for multiple disk addressings during row writing, resulting in low write efficiency compared to traditional row-oriented databases.
Innovation Solution
Storing same-row data in the same disk cylinder, with data in each section of columns stored successively, allowing a magnetic disk to complete row writing with only one rotation, and using hardware marking to align standard columns across platters for parallel reading and writing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If columns are stored separately after division in a columnar database, then query speed is improved by reading only necessary columns, but write speed deteriorates due to multiple disk addressings required during row writing
Solution Approach 1:
The patent segments data storage into two distinct layouts: columnar storage for query operations and row-oriented temporary storage for write operations. During writing, data is temporarily organized by row in memory, allowing sequential disk writing without multiple addressings. After writing completes, the data is reorganized into the columnar layout. This segmentation resolves the contradiction by providing optimized storage structures for each operation type at different times.
Solution Approach 2:
The patent performs preliminary organization of data in memory before writing to disk. All row data is prepared and organized in memory first, then written to disk in a single sequential operation. This preliminary action eliminates the need for multiple disk addressings during the write operation, thereby improving write speed while maintaining the columnar storage structure for subsequent queries.
2Stability of the object's composition
If data is written in random order to match column positions, then columnar storage structure is maintained, but write efficiency deteriorates compared to sequential writing
Solution Approach 1:
The patent dynamically switches between two storage layouts based on the operation type. For write operations, data is temporarily organized in a row-oriented dynamic structure in memory, allowing sequential writing to disk. For query operations, the data exists in the static columnar structure on disk. This dynamic adaptation resolves the contradiction by allowing the system to optimize for write efficiency during writing while maintaining columnar structure for queries.
Solution Approach 2:
The patent introduces memory as an intermediary between the row-oriented input data and the columnar disk storage. Data is first organized in memory with row-oriented structure, then written sequentially to disk. The memory intermediary buffers the transformation from row-oriented to columnar layout, allowing sequential writing without compromising the final columnar structure on disk.
3Manufacturing precision
If multiple disk addressings are performed during row writing to place data in correct column positions, then data is stored in correct locations, but write time increases significantly
Solution Approach 1:
The patent performs preliminary organization of all row data in memory before the actual disk writing operation. Each row's data is pre-positioned in memory according to its final columnar destination. This preliminary action ensures that when data is written to disk, it can be written sequentially without requiring multiple addressings, thereby maintaining data placement accuracy while significantly reducing write time.
Solution Approach 2:
The patent replaces the mechanical disk seeking and positioning system with an in-memory organization system. Instead of using the disk's physical addressing mechanism to place data in correct column positions, all positioning is done in memory using pointer manipulation. The disk then receives data in sequential order, eliminating the time-consuming mechanical seeking operations while maintaining precise data placement.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of the present invention disclose a data processing method, including: acquiring to-be-stored data in a data table in a columnar database; dividing and/or combining the to-be-stored data according to space of an allocated disk cylinder; and storing same-row data of the to-be-stored data in a same disk cylinder, and in the disk cylinder, successively storing data in each section of columns. Correspondingly, the present invention further discloses a data writing method and a related apparatus and device. A technical problem in the prior art that a write speed is slow when a columnar database is running is resolved by using the present invention, which greatly improves write efficiency of a columnar database. In addition, direct parallel reading or writing may also be implemented by means of hardware marking.