Page-Loadable Column Architecture for In-Memory Data Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern in-memory database systems face challenges in efficiently managing large datasets that combine business data with IoT and social media data, requiring high performance while optimizing precious main memory usage for low-value or low-density data.
Innovation Solution
The implementation of a page-loadable column architecture in an in-memory database system, which includes a unified table with read-optimized and write-optimized sections, uses delta-merge operations and dynamically rebuilds non-critical data structures based on query requirements to reduce memory footprint and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all data is loaded into main memory for high-performance analytical workloads, then query performance is improved, but memory usage increases and precious main memory is wasted on low-value or low-density data
Solution Approach 1:
The columnar data structure is segmented into multiple pages, with only the necessary pages loaded into main memory based on query requirements. This allows the system to maintain high query performance for frequently accessed data while reducing overall memory usage by keeping less critical data in storage rather than main memory.
Solution Approach 2:
The system dynamically loads and evicts column pages based on query patterns and data access requirements. The memory management becomes adaptive, loading only the necessary data into main memory when queries require it, rather than continuously maintaining all data in memory, thus optimizing the balance between performance and memory usage.
2Speed
If data structures are rebuilt to optimize for current query requirements, then query performance is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs partial rebuilding of only the necessary data structures and pages rather than completely rebuilding the entire columnar store. This selective approach reduces the computational overhead and time required while still optimizing the critical paths for current query workloads.
Solution Approach 2:
The system pre-loads and pre-processes data structures and index pages that are likely to be needed based on query patterns, rather than waiting for queries to trigger full rebuilds. This preliminary action reduces the impact of rebuilding operations on query response time.
3Quantity of substance
If page-loadable columns are implemented to reduce memory footprint, then memory usage is optimized, but system complexity increases
Solution Approach 1:
The page-loadable column architecture uses universal data structures and standardized page management mechanisms that can handle different data types and access patterns through a unified approach. This reduces the actual complexity increase by providing a single framework that serves multiple purposes rather than requiring separate mechanisms for different operations.
Data Source
AI summary
Cracking page-loadable columns for in-memory data management is described herein. An embodiment operates by accessing a column according to a received query, determining that the received query requires a non-critical data structure associated with the column, and rebuilding the non-critical data structure from data associated with the column.