Hybrid Space Allocation for Large Write-Once Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Relational database management systems face challenges in efficiently managing large amounts of data due to the memory overhead associated with converter maps in logical space allocation techniques, leading to prohibitively large memory requirements and prolonged initialization times, especially when dealing with hundreds of terabytes of data.
Innovation Solution
Implementing a hybrid space allocation module that uses both logical and physical allocation schemes, allocating space in two sizes (small and big pages), with a converter module managing logical page addresses and a big object directory manager handling physical allocation, allowing for efficient storage of large data sets without the need for reorganization operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If logical space allocation with converter maps is used to manage database pages, then reorganization operations are not necessary and space is automatically reclaimed, but memory overhead becomes prohibitively large and initialization time increases for hundreds of terabytes of data
Solution Approach 1:
The patent segments the database management system into two distinct space allocation subsystems: a logical space allocation subsystem using converter maps for small pages, and a physical space allocation subsystem using free block managers for large pages. This segmentation allows each subsystem to handle different data sizes with appropriate allocation strategies, reducing the memory overhead of converter maps while maintaining automatic space reclamation for frequently accessed data through the logical allocation path.
Solution Approach 2:
The patent applies different allocation strategies to different parts of the database based on local requirements: small pages (4KB-64KB) use logical allocation with converter maps for fast automatic reclamation, while large pages (1MB-1GB) use physical allocation with free block managers for memory efficiency. This local quality approach optimizes both memory usage and operational ease for different data access patterns.
2Adaptability or versatility
If converter maps are used to map logical page addresses to physical page addresses, then logical space allocation is enabled, but the converter map size becomes prohibitively large for hundreds of terabytes of data
Solution Approach 1:
The patent segments the address mapping function by creating separate mapping mechanisms for different page sizes: converter maps handle small page logical-to-physical mapping, while a simplified mapping approach handles large page allocation. This segmentation reduces the converter map size by excluding large pages from its management scope.
Solution Approach 2:
The patent introduces a new dimension of page size classification, dividing database pages into small pages and large pages. This dimensional change allows the system to apply different mapping strategies appropriate to each dimension, reducing the complexity of the converter map by only managing small page mappings.
3Productivity
If large page sizes are used to store extremely large data sets, then storage efficiency improves, but reorganization freedom is lost with physical allocation
Solution Approach 1:
The patent segments data storage into two paths: small pages with logical allocation for reorganization freedom, and large pages with physical allocation for storage efficiency. By segmenting the storage system this way, the patent achieves both reorganization freedom for frequently accessed data and storage efficiency for large data sets.
Solution Approach 2:
The patent applies local quality by matching allocation strategies to data characteristics: logical allocation with reorganization freedom for small, frequently accessed pages, and physical allocation with high storage efficiency for large, less frequently modified pages. This local optimization achieves both goals simultaneously.
Data Source
AI summary
Methods and systems for managing large write-once tables are described. In some embodiments, a relational database management system includes a space allocation module that utilizes both a logical space allocation scheme, as well as a physical space allocation scheme, to allocate space in units (e.g., pages) having two different sizes—small pages and big pages. For instance, small pages are logically allocated with a conventional converter module, which manages a converter table for mapping logical pages to physical pages, while big pages are physically allocated with an object directory manager, which manages big objects comprised of big pages.