Columnar Database Write Optimization via Disk Cylinder Alignment

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Columnar databases experience slow write speeds due to the need for multiple disk addressings during row writing, resulting in low write efficiency compared to traditional row-oriented databases.

Innovation Solution

Storing same-row data in the same disk cylinder, with data in each section of columns stored successively, allowing a magnetic disk to complete row writing with only one rotation, and using hardware marking to align standard columns across platters for parallel reading and writing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If columns are stored separately after division in a columnar database, then query speed is improved by reading only necessary columns, but write speed deteriorates due to multiple disk addressings required during row writing

Engineering Contradiction:
Improvequery speedVSAvoidwrite speed
Core Design Contradiction:
SpeedVSProductivity

Solution Approach 1:

The patent segments data storage into two distinct layouts: columnar storage for query operations and row-oriented temporary storage for write operations. During writing, data is temporarily organized by row in memory, allowing sequential disk writing without multiple addressings. After writing completes, the data is reorganized into the columnar layout. This segmentation resolves the contradiction by providing optimized storage structures for each operation type at different times.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary organization of data in memory before writing to disk. All row data is prepared and organized in memory first, then written to disk in a single sequential operation. This preliminary action eliminates the need for multiple disk addressings during the write operation, thereby improving write speed while maintaining the columnar storage structure for subsequent queries.

Inventive Principle:
Principle #10Preliminary action

2Stability of the object's composition

If data is written in random order to match column positions, then columnar storage structure is maintained, but write efficiency deteriorates compared to sequential writing

Engineering Contradiction:
Improvecolumnar storage structureVSAvoidwrite efficiency
Core Design Contradiction:
Stability of the object's compositionVSProductivity

Solution Approach 1:

The patent dynamically switches between two storage layouts based on the operation type. For write operations, data is temporarily organized in a row-oriented dynamic structure in memory, allowing sequential writing to disk. For query operations, the data exists in the static columnar structure on disk. This dynamic adaptation resolves the contradiction by allowing the system to optimize for write efficiency during writing while maintaining columnar structure for queries.

Inventive Principle:
Principle #15Dynamics

Solution Approach 2:

The patent introduces memory as an intermediary between the row-oriented input data and the columnar disk storage. Data is first organized in memory with row-oriented structure, then written sequentially to disk. The memory intermediary buffers the transformation from row-oriented to columnar layout, allowing sequential writing without compromising the final columnar structure on disk.

Inventive Principle:
Principle #24Intermediary (Mediator)

3Manufacturing precision

If multiple disk addressings are performed during row writing to place data in correct column positions, then data is stored in correct locations, but write time increases significantly

Engineering Contradiction:
Improvedata placement accuracyVSAvoidwrite time
Core Design Contradiction:
Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary organization of all row data in memory before the actual disk writing operation. Each row's data is pre-positioned in memory according to its final columnar destination. This preliminary action ensures that when data is written to disk, it can be written sequentially without requiring multiple addressings, thereby maintaining data placement accuracy while significantly reducing write time.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent replaces the mechanical disk seeking and positioning system with an in-memory organization system. Instead of using the disk's physical addressing mechanism to place data in correct column positions, all positioning is done in memory using pointer manipulation. The disk then receives data in sequential order, eliminating the time-consuming mechanical seeking operations while maintaining precise data placement.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentEP3093772B1Data processing method, data writing method, and related apparatus
Publication Date: 2018.07.18 HUAWEI TECH CO LTD
  • EP3093772B1 patent drawingFigure 1
  • EP3093772B1 patent drawingFigure 2
  • EP3093772B1 patent drawingFigure 3~4

AI summary

Embodiments of the present invention disclose a data processing method, including: acquiring to-be-stored data in a data table in a columnar database; dividing and/or combining the to-be-stored data according to space of an allocated disk cylinder; and storing same-row data of the to-be-stored data in a same disk cylinder, and in the disk cylinder, successively storing data in each section of columns. Correspondingly, the present invention further discloses a data writing method and a related apparatus and device. A technical problem in the prior art that a write speed is slow when a columnar database is running is resolved by using the present invention, which greatly improves write efficiency of a columnar database. In addition, direct parallel reading or writing may also be implemented by means of hardware marking.