In-Memory Database Indexing for Predictive Column Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory columnar databases experience performance degradation due to the dynamic loading of reports, which is proportional to the number and size of columns, leading to inefficient memory management and system overload.
Innovation Solution
An intelligent data management system utilizing machine learning to predict which data should be loaded and unloaded from memory based on user behavior patterns, optimizing memory usage by dynamically adjusting data access according to user roles and tasks, and leveraging a graph cube network for data relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all columns are loaded into memory when a report is accessed, then data availability is improved, but memory overload and system performance degradation occur
Solution Approach 1:
The patent segments the column data into multiple partitions or chunks that can be independently managed. Instead of loading all columns at once, the system divides the data storage into manageable segments that can be loaded selectively based on query requirements, thus improving data availability while controlling memory usage.
Solution Approach 2:
The system performs preliminary indexing and metadata preparation work before actual data loading. By pre-processing and organizing column metadata, the system can quickly determine which specific column segments need to be loaded, enabling efficient selective loading rather than loading all data, thus balancing data availability with memory constraints.
2Quantity of substance
If lazy loading is used to avoid memory overload, then memory management is improved, but system performance degrades due to dynamic loading overhead
Solution Approach 1:
The system performs preliminary indexing and metadata organization before actual data loading occurs. This pre-processing work enables the lazy loading mechanism to quickly identify and load only the necessary column segments when needed, reducing the overhead of dynamic loading while maintaining memory management efficiency.
Solution Approach 2:
The patent replaces the traditional mechanical approach of loading entire tables or all columns with a smarter system that uses metadata-driven selective loading. The metadata index acts as a replacement for brute-force data loading, enabling the system to efficiently determine which specific column segments to load based on query requirements, thus improving performance while maintaining memory efficiency.
3Speed
If columnar storage is used for analytical solutions, then query performance is improved, but complexity of data management increases
Solution Approach 1:
The patent segments columnar data into manageable partitions with associated metadata indexes. This segmentation approach maintains the query performance benefits of columnar storage by keeping related data together, while simultaneously reducing management complexity by organizing data into smaller, more controllable units that can be independently loaded and managed.
Solution Approach 2:
The system introduces metadata indexes as an intermediary layer between the query processor and the columnar data storage. This metadata layer simplifies data management by providing a simplified interface for querying and loading data, while the underlying columnar storage maintains its performance advantages. The metadata acts as a mediator that handles the complexity of managing segmented columnar data.
Data Source
AI summary
Systems and techniques for intelligent data management are described herein. An example technique may include receiving a set of training data from a graph cube server, training a machine learning model using the received set of training data, and determining, using the machine learning model, a data usage pattern. The example technique may include receiving an indication that a first user of a user group logged into an intelligent data management system, the indication including a first user detail of the first user, and loading a first column into memory based on the determined data usage pattern and the first user detail.