Columnar Database Complex Data Type Encoding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Many existing columnar databases do not support complex data types, making them less effective in processing data from sources like web traffic, warehouse management, and IoT, which store sparse, multidimensional, and semi-structured data.

Innovation Solution

Encoding complex data types over table columns in columnar databases by mapping their fields to columns with primitive data types, allowing for optimized query generation and late-materialization processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If columnar databases use only primitive data types, then storage and query processing are simple and efficient, but they cannot effectively process complex data types from sources like web traffic, warehouse management, and IoT

Engineering Contradiction:

Improvesupport for complex data typesVSAvoiddata type structure

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The complex data type is segmented into multiple primitive data type fields. Each field of the complex data type (e.g., structure, array, map) is mapped to a separate column in the columnar database table, allowing the database to process complex data through its existing primitive data type infrastructure while supporting complex data structures.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary encoding layer that maps complex data types to primitive data types. This intermediary mapping allows the columnar database to handle complex data types without requiring native support for them, by translating complex structures into combinations of primitive types that the database can efficiently process.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If complex data types are natively supported in columnar databases, then processing capability improves, but query optimization and materialization efficiency deteriorate

Engineering Contradiction:

Improvequery processing capabilityVSAvoidmaterialization time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

By segmenting complex data types into primitive fields stored in separate columns, the system enables selective materialization of only the specific columns needed for a given query. This avoids the need to materialize entire complex data structures, reducing materialization time while maintaining full processing capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by allowing different columns (representing different fields of complex data types) to be materialized independently based on query requirements. This enables optimized query generation that materializes only the necessary portions of complex data structures, improving efficiency by avoiding unnecessary materialization overhead.

Inventive Principle:

Principle #3Local quality

3Productivity

If complex data types are encoded over multiple columns, then processing efficiency improves, but data organization and retrieval complexity increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoiddata organization structure

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent creates a universal mapping framework that works across all complex data types (structures, arrays, maps) by treating them uniformly as collections of primitive fields. This multi-functional approach allows the same columnar storage and query optimization mechanisms to handle diverse complex data types, improving processing efficiency without requiring separate handling logic for each data type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system creates a simplified copy or representation of complex data types using only primitive data types in the columnar format. This copied representation maintains the essential structure and semantics of the original complex data types while being fully compatible with the columnar database's efficient processing mechanisms, thereby improving processing efficiency without increasing actual storage complexity.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS11327995B2Complex data type encoding within columnar database

Publication Date: 2022.05.10 ROCKET SOFTWARE

AI summary

A complex data type is encoded over columns of a table of a columnar database by mapping fields of the complex data type to the columns. An optimized query can be generated for a query specifying the complex data type. The optimized query specifies the columns to which the fields specified within the query are mapped, instead of specifying the fields. The optimized query can be processed against the database in a late materialization manner to fulfill the query.