Distributed Database Partitioned Dataset Storage Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in storing and managing large datasets due to limitations in scalability, data integrity, and performance, particularly in handling multiple data portions across partitions, which can lead to issues with data consistency and concurrency during read and write operations.
Innovation Solution
A distributed database system is implemented with a segmentation approach where each dataset is divided into a primary and zero or more additional data portions, with metadata in the primary partition mapping the locations of additional portions, allowing concurrent access and atomic updates across multiple partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If datasets are segmented into multiple data portions stored in respective partitions, then storage capacity and scalability are improved, but data consistency and concurrency control become more complex
Solution Approach 1:
The patent divides datasets into multiple data portions (primary and secondary) that can be stored in separate partitions across different storage nodes. This segmentation enables horizontal scaling of storage capacity while maintaining the ability to reconstruct complete datasets by combining portions from multiple partitions.
Solution Approach 2:
The patent introduces metadata as an intermediary component that stores information about the locations and relationships of data portions. This metadata acts as a mediator that simplifies the management of segmented data, enabling efficient retrieval and consistency control without requiring complex coordination between all storage nodes.
2Quantity of substance
If datasets are segmented into multiple data portions stored in respective partitions, then storage scalability is improved, but access time and latency increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing metadata that contains information about data portion locations and relationships. This preliminary organization of information enables direct access to required data portions without requiring sequential searching or complex coordination, thereby reducing access latency despite data being distributed across multiple partitions.
3Productivity
If metadata maps all additional data portions to secondary partitions, then data retrieval efficiency is improved, but memory usage and storage overhead increase
Solution Approach 1:
The patent applies local quality by storing metadata locally with each primary data portion rather than maintaining a centralized metadata repository. This distributed metadata approach enables each partition to independently manage its data portions, improving retrieval efficiency while distributing the storage overhead burden across multiple nodes rather than concentrating it in a single location.
Data Source
AI summary
Disclosed herein is a system, comprising a distributed database comprising a plurality of partitions allocated in memory for storing a plurality of datasets. Wherein each dataset stored in the distributed database comprises at least a primary data portion stored in a respective primary partition of the plurality of partitions and at least zero additional data portions stored in respective secondary partitions of the plurality of partitions. Wherein each data portion of each dataset is mapped to a respective partition in which it is stored by a respective identifier. Wherein the primary data portion of each dataset comprises metadata stored in the primary partition, the metadata is indicative of the identifier of all additional data partitions of the respective dataset.