Distributed In-Memory Database Partitioning for Storage Capacity and Access Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently storing and retrieving large amounts of data at high speeds due to limitations in memory size, leading to slow access times when using conventional disk storage.
Innovation Solution
An in-memory database system with a distributed computing architecture that compresses data and uses a combination of software modules such as a partitioner, search manager, analytics agent, and search conductor to manage and query data efficiently, allowing for fault-tolerant and scalable data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in disk storage to handle large database scales, then storage capacity is sufficient, but data access speed becomes slow
Solution Approach 1:
The patent segments the database into multiple partitions that are distributed across different nodes in a cluster. Each partition can be stored in memory or on disk independently, allowing parallel access to different data segments. This segmentation enables the system to provide fast in-memory access for frequently queried partitions while maintaining adequate storage capacity across the distributed cluster.
Solution Approach 2:
The patent transitions from a single-dimensional storage hierarchy (disk vs. memory) to a multi-dimensional distributed architecture. Data is organized across multiple dimensions: spatial distribution across cluster nodes, hierarchical organization into partitions and collections, and selective placement in memory or disk based on access patterns. This multi-dimensional approach allows simultaneous optimization of speed and storage capacity.
2Speed
If all data is stored in memory to achieve fast access, then data access speed is improved, but memory size limitations prevent handling large databases
Solution Approach 1:
The patent applies local quality by allowing different partitions or collections within the database to have different storage characteristics. Frequently accessed data can be placed in memory on specific nodes for fast access, while less frequently accessed data can be stored on disk. Each node can independently manage its memory and disk resources based on local access patterns, optimizing the balance between speed and capacity at each location.
Solution Approach 2:
The system dynamically changes storage parameters based on data characteristics and access patterns. Data can be migrated between memory and disk storage based on usage frequency, and the degree of compression can be adjusted based on data types and access requirements. These parameter changes allow the system to adapt to varying workload demands while maintaining fast access for critical data.
3Quantity of substance
If a distributed cluster architecture is used to handle large data scales, then storage capacity and processing power are improved, but system complexity increases
Solution Approach 1:
The patent implements universal node designs where each cluster node can perform multiple functions: storing data partitions, processing queries, performing aggregations, and managing local memory-disk operations. This multi-functionality reduces the need for specialized components and simplifies the overall system architecture compared to having separate storage nodes and compute nodes, while still providing the benefits of distributed scaling.
Solution Approach 2:
The distributed cluster implements self-service mechanisms where nodes automatically perform load balancing, data distribution, and failure recovery without centralized coordination for every operation. The system self-manages partition placement, replica distribution, and query routing across the cluster, reducing the operational complexity burden on administrators while maintaining robust distributed capabilities.
4Quantity of substance
If data compression is applied to increase storage efficiency, then storage capacity is improved, but data retrieval and processing speed may be reduced
Solution Approach 1:
The patent applies partial compression selectively to specific partitions, collections, or even individual fields based on their characteristics and access patterns. Frequently accessed data may be stored with minimal or no compression to maximize retrieval speed, while less frequently accessed data can be heavily compressed to optimize storage utilization. This partial application of compression allows the system to gain storage efficiency without significantly impacting the performance of critical data access operations.
Data Source
AI summary
An in-memory database system and method for administrating a distributed in-memory database, comprising one or more nodes having modules configured to store and distribute database partitions of collections partitioned by a partitioner associated with a search conductor. Database collections are partitioned according to a schema. Partitions, collections, and records, are updated and removed when requested by a system interface, according to the schema. Supervisors determine a node status based on a heartbeat signal received from each node. Users can send queries through a system interface to search managers. Search managers apply a field processing technique, forward the search query to search conductors, and return a set of result records to the analytics agents. Analytics agents perform analytics processing on a candidate results records from a search manager. The search conductors comprising partitioners associated with a collection, search and score the records in a partition, then return a set of candidate result records after receiving a search query from a search manager.