Native Primitive Copy for In-Memory and Paged Data Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently handling large data volumes due to memory limitations, particularly during operations involving in-memory databases, where copying data between different formats like PagedBitVector and BitVector is time-consuming and resource-intensive.
Innovation Solution
Implementing native copy techniques that copy data one page at a time, ensuring byte compatibility, and utilizing page prefetching to optimize the process, reducing processing time and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is copied between different formats (PagedBitVector and BitVector) using traditional methods, then data can be transferred between storage and memory, but the operation is time-consuming and resource-intensive
Solution Approach 1:
The patent segments the data copying process into page-level operations. Instead of copying entire data structures at once, the system divides data into pages and copies them individually using native copy operations. This segmentation enables more efficient memory management and reduces the overall copying time by allowing parallel processing and better utilization of memory bandwidth.
Solution Approach 2:
The patent implements a native copy mechanism that creates direct byte-compatible copies of data between persistence store and in-memory store. The copy operation preserves the exact bit representation of primitives during transfer, eliminating the need for decoding and encoding operations. This direct copying approach significantly reduces processing time and computational resources compared to traditional format conversion methods.
2Productivity
If traditional data copying methods are used, then data can be transferred between formats, but processing resources are consumed excessively
Solution Approach 1:
The native copy mechanism performs direct byte-compatible copying of primitive data structures between persistence store and in-memory store without format conversion. This eliminates decoding and encoding operations, significantly reducing CPU cycles and processing resource consumption while maintaining data integrity.
Solution Approach 2:
The patent changes the parameter of data representation compatibility by ensuring that the bit representation of primitives in the persistence store matches the bit representation expected by the in-memory store. This parameter alignment enables direct copying operations without resource-intensive format conversion, improving processing efficiency.
3Speed
If data is loaded into memory for in-memory operations, then query execution speed improves, but memory limitations prevent loading all required data
Solution Approach 1:
The patent segments large data sets into manageable pages that can be selectively loaded into memory based on query requirements. Instead of loading entire data structures, the system loads only the necessary pages into memory, enabling fast query execution on relevant data while preserving the ability to handle large data volumes that exceed available memory capacity.
Data Source
AI summary
A database system detects a query targeting a first database object stored in a unified persistence format in a persistence store. In response to detecting the query, the database system determines whether to load the first database object in column-loadable format or page-loadable format into an in-memory store. In response to determining to load the first database object in the column-loadable format, the database system determines if a first bit representation of one or more corresponding primitives in the persistence store is compatible with a second bit representation used for storing the primitives in the in-memory store. The database system performs a native copy operation to copy the primitives from the persistence store to the in-memory store if the bit representations are compatible, where the native copy operation involves copying the primitives on a page-by-page basis without decoding and encoding values of the one or more primitives.