Unbalanced Database Partitioning for Scalable Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In shared nothing database architectures, scaling single-node applications to multi-node systems is challenging due to inefficient data partitioning, leading to redundant storage and high overhead in maintaining database integrity, especially when performing operations that require access to both master and document data.
Innovation Solution
Implementing an unbalanced partitioning scheme where shard identifiers are used to map data to specific partitions, allowing for faster data access and reduced overhead by eliminating the need for exhaustive searches across all partitions, with unique ranges of data identifiers assigned to each shard, enabling efficient storage and retrieval of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is equally distributed over multiple database nodes in a shared nothing architecture, then scalability is improved, but resource consumption increases due to exhaustive searches across all partitions
Solution Approach 1:
The patent segments the database into distinct partitions (master data partitions and document data partitions) that can be independently managed and searched. This segmentation allows the system to scale by adding more partitions without requiring exhaustive searches across all data, as the search space is divided into manageable segments that can be processed in parallel or selectively accessed.
Solution Approach 2:
The patent applies local quality by assigning specific characteristics to different partitions - master data partitions store reference information while document data partitions store detailed content. This allows optimization of storage and access patterns for each partition type, improving overall system efficiency while maintaining scalability through targeted data placement and selective querying.
2Productivity
If data is stored in a shared everything architecture, then resource utilization is improved, but redundant storage and overhead for maintaining database integrity increase
Solution Approach 1:
The patent divides the shared everything architecture into separate master data partitions and document data partitions, eliminating redundancy by ensuring each data type is stored in its designated partition. This segmentation prevents duplicate storage of the same data across multiple nodes while still allowing high resource utilization through parallel access to different partition types.
Solution Approach 2:
The patent extracts document data from master data partitions and stores them in separate document data partitions. This extraction eliminates the need to store duplicate copies of document data in multiple locations, reducing redundant storage while maintaining the ability to efficiently access both master and document data through the partitioned structure.
3Reliability
If exhaustive searches are performed across all database partitions, then data integrity is maintained, but operation time increases
Solution Approach 1:
The patent segments the search space into master data partitions and document data partitions, allowing the database to perform targeted searches within specific partitions rather than exhaustive searches across all data. This maintains data integrity by ensuring complete coverage of relevant partitions while significantly reducing operation time through eliminated redundant searches.
Solution Approach 2:
The patent implements preliminary action by establishing clear partitioning structures and data placement rules before data operations occur. This pre-organization of data into master and document partitions enables the system to predictably determine which partitions to search without performing exhaustive searches, maintaining integrity while reducing operation time through pre-planned data access paths.
Data Source
AI summary
Provided is a database system and method in which storage is partitioned in an unbalanced format for faster access. In one example, the method may include one or more of receiving a request to store a data record, identifying a partition from among a plurality of partitions of a database based on a shard identifier in the request, automatically determining a unique range of data identifiers designated to the partition from the plurality of partitions, respectively, based on an unbalanced partitioning, determining whether the data identifier is available within the unique range of data identifiers of the identified partition, and storing the data record at the identified partition in response to determining the data identifier is available. The unbalanced partitioning according to various embodiments reduces the partitions that need to be checked during a data insert/access operation of the database.