Multi-Dimensional Database Partitioning for Unstructured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database partitioning methods, particularly in SIEM systems, are inefficient as they primarily rely on time-based partitioning, leading to performance issues when handling large volumes of data and requiring extensive resource overhead, especially in multi-tenant environments where data is unstructured and varies significantly in format.
Innovation Solution
Implementing a database system that partitions data using multiple dimensions, such as tenant and variant identifiers, allowing for more efficient organization and retrieval by creating physical partitions that act as a first-level filter, reducing the need for scanning multiple partitions and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If time-based partitioning is used to organize large volumes of data, then data organization is simplified, but data retrieval performance deteriorates and resource overhead increases
Solution Approach 1:
The patent transitions from single-dimension (time-based) partitioning to multi-dimensional partitioning by introducing tenant identifiers and variant identifiers as additional partitioning dimensions. This creates a hierarchical partitioning structure where data is first divided by tenant, then by variant, and finally by time period, enabling more precise data location and retrieval without scanning entire partitions.
2Ease of operation
If single-value partitioning is used to simplify partition management, then partition creation is easier, but data retrieval efficiency deteriorates
Solution Approach 1:
The patent segments the partitioning strategy into multiple hierarchical levels: tenant-level partitions, variant-level partitions within tenants, and time-based partitions within variants. This segmentation allows the system to manage complexity through organized hierarchy while achieving efficient data retrieval by narrowing the search scope at each level.
3Productivity
If extensive indexing is used to improve data retrieval, then query performance improves, but resource overhead increases
Solution Approach 1:
The patent performs preliminary data organization by creating tenant-based and variant-based partitions before time-based partitioning. This preliminary structuring enables the system to quickly identify and access only relevant tenant and variant partitions without requiring extensive indexing mechanisms, thereby reducing resource overhead while maintaining query performance.
Data Source
AI summary
Methods, apparatus, systems and articles of manufacture to partition a database are disclosed. An example apparatus includes a dimension identifier to identify at least two dimensioning values associated with a received query, the query including unstructured data to be written to a database. A partition manager is to select a partition into which data is to be written based on the at least two dimensioning values. At least one of the at least two dimensioning values being a variant of unstructured data associated with the query. A partition creator is to, in response to the selected partition not existing in the database, create the selected partition. A data writer is to write the data to the selected partition.