In-Memory Object Sharding Across NUMA Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in query processing due to data being stored in row-major disk blocks, which are not optimal for in-memory queries, and migrating to column-major formats is complex, leading to performance issues and overhead in maintaining column store indexes or database replication.
Innovation Solution
The system maintains data in both row-major disk blocks and a mirror format that is column-major, allowing for efficient query processing by sharding data across NUMA nodes and using in-memory compression units, with transactional consistency and compression to optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in row-major disk blocks and cached in buffer cache, then data persistence and row-based operations are maintained, but query processing performance for column-based operations is suboptimal
Solution Approach 1:
The patent segments data storage into two distinct formats: row-major disk blocks for persistence and column-major in-memory structures for query processing. This segmentation allows each format to optimize for its specific use case without requiring full migration or complex hybrid structures
Solution Approach 2:
The patent introduces column store indexes as an intermediary structure that bridges row-major stored data and column-based query operations. These indexes are maintained separately from the primary row-major tables, enabling efficient columnar access patterns without changing the underlying storage format
2Productivity
If column store indexes are added to maintain row-major tables, then column-based query efficiency is improved, but maintenance overhead increases
Solution Approach 1:
The patent implements column store indexes selectively for specific columns that benefit most from columnar access patterns, rather than creating indexes for all columns. This partial action approach reduces maintenance overhead while still providing significant performance benefits for key query workloads
3Productivity
If database replication is used to provide both row-major and column-major formats, then query optimization is enabled, but replication lag and consistency issues occur
Solution Approach 1:
The patent merges row-major and column-major data formats within a single database instance rather than using separate replicated systems. The column store indexes are maintained as part of the same database system, ensuring immediate consistency between row-major tables and columnar indexes without replication lag
Data Source
AI summary
Techniques are provided for sharding objects across different compute nodes. In one embodiment, a database server instance generates, for an object, a plurality of in-memory chunks including a first in-memory chunk and a second in-memory chunk, where each in-memory chunk includes a different portion of the object. The database server instance assigns each in-memory chunk to one of a plurality of computer nodes including the first in-memory chunk to a first compute node and a second in-memory chunk to a second local memory of a second compute node. The database server instance stores an in-memory map that indicates a memory location for each in-memory chunk. The in-memory map indicates that the first in-memory chunk is located in the first local memory of the first compute node and that the second in-memory chunk is located in the second local memory of the second compute node.