Multi-dimensional Partitioning Storage Layer for Distributed Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current database systems face inefficiencies in handling ad hoc queries, particularly in petabyte-scale databases, due to limited support for multiple partitioning and sorting orders, leading to high resource consumption and slow query performance, as existing solutions require duplicating data and increasing storage and maintenance costs.

Innovation Solution

The implementation of a storage layer that allows for multiple simultaneous sorting and partitioning strategies without additional storage or performance costs, using logical replication and super-extents, which enables efficient data access and recovery while maintaining high availability and reliability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If data is partitioned and sorted by one column to speed up queries, then query performance for that column is improved, but the system cannot efficiently handle queries on other columns without additional partitioning

Engineering Contradiction:
Improvequery performanceVSAvoidsupport for multiple partitioning orders
Core Design Contradiction:
SpeedVSAdaptability or versatility

Solution Approach 1:

The patent segments the data storage into multiple sorted representations (primary sort order and secondary sort order) within the same data structure. Each segment maintains different sorting characteristics, allowing the system to serve queries on different columns efficiently without requiring separate physical partitions for each sort order.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent creates a universal data storage structure that serves multiple query patterns simultaneously. By incorporating both primary and secondary sort orders in a single storage system, the database can handle queries on either column efficiently, making the storage system multi-functional rather than specialized for a single access pattern.

Inventive Principle:
Principle #6Universality (Multi-functionality)

2Speed

If data is duplicated to support multiple partitioning orders, then queries on different columns can be optimized, but storage space and maintenance complexity increase significantly

Engineering Contradiction:
Improvequery performance for multiple columnsVSAvoidstorage space
Core Design Contradiction:
SpeedVSQuantity of substance

Solution Approach 1:

The patent merges multiple sorted representations into a single data storage structure. Instead of creating separate physical copies of data for different sort orders, the system combines primary and secondary sort orders within the same storage framework, eliminating redundant storage while maintaining query performance benefits.

Inventive Principle:
Principle #5Merging (Combining)

Solution Approach 2:

The patent adds a secondary sort order dimension to the existing primary sort order structure. Rather than duplicating data in separate physical locations, the system organizes data along multiple sorting dimensions simultaneously, allowing efficient access from either dimension without increasing storage requirements.

Inventive Principle:
Principle #17Another dimension (Dimensionality change)

3Speed

If data is duplicated to support multiple partitioning orders, then queries on different columns can be optimized, but system complexity and consistency maintenance costs increase

Engineering Contradiction:
Improvequery performanceVSAvoidcomplexity of keeping copies consistent
Core Design Contradiction:
SpeedVSDevice complexity

Solution Approach 1:

The patent merges multiple sorted representations into a single data storage structure. Instead of creating separate physical copies of data for different sort orders, the system combines primary and secondary sort orders within the same storage framework, eliminating redundant storage while maintaining query performance benefits.

Inventive Principle:
Principle #5Merging (Combining)

4Adaptability or versatility

If the entire database is scanned for ad hoc queries, then all data can be examined, but resource consumption and query execution time increase dramatically

Engineering Contradiction:
Improveability to handle any query typeVSAvoidresource consumption
Core Design Contradiction:
Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent segments the data storage into multiple sorted representations (primary sort order and secondary sort order) within the same data structure. Each segment maintains different sorting characteristics, allowing the system to serve queries on different columns efficiently without requiring separate physical partitions for each sort order.

Inventive Principle:
Principle #1Segmentation

Data Source

PatentUS10810206B2Efficient multi-dimensional partitioning and sorting in large-scale distributed data processing systems
Publication Date: 2020.10.20 MICROSOFT TECHNOLOGY LICENSING LLC
  • US10810206B2 patent drawing
  • US10810206B2 patent drawing
  • US10810206B2 patent drawing

AI summary

Methods, systems, and computer programs are presented for structuring a database to support multiple partitioning orders at the storage layer. One method includes an operation for identifying partitioning fields for a database that is stored distributed across computing devices, where each computing device stores an extent that holds a subset of entries from the database. For each partitioning field, the database entries are stored in extents associated with the partitioning field, the database entries in the extents for the partitioning field being organized based on the value of the partitioning field. Further, the method includes operations for receiving a database query that includes a filter based on values of a selected partitioning field, and for retrieving the data for the database query from one or more of the extents associated with the selected partitioning field. The retrieved data is returned for the database query.