Auxiliary Storage for Large Column Values in Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional columnar databases face inefficiencies in handling large column values, leading to memory bloat during writing and reading operations, and restrictive size limitations that fail to meet user storage needs.
Innovation Solution
A database system comprising a write storage, a read storage, and an auxiliary storage, where large column values are stored in the auxiliary storage after compression and division into chunks, with associating data stored in the write or read storage to efficiently manage and retrieve these values, minimizing memory usage and optimizing storage capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple column values are compressed and stored together in a row group, then columnar storage efficiency is improved, but memory bloat occurs during writing and reading operations
Solution Approach 1:
The patent segments large column values into separate storage locations (auxiliary storage) rather than storing them within row groups. Each large column value is stored as an independent object in auxiliary storage, and only references are stored in the row group, preventing memory bloat while maintaining columnar storage efficiency for smaller values.
2Quantity of substance
If a limit on the number of rows in a row group is imposed, then memory usage during writing and reading is controlled, but the benefit of columnar storage is significantly reduced
Solution Approach 1:
The patent extracts large column values from the row group storage structure and places them in auxiliary storage. This allows row groups to maintain their columnar organization for small values while large values are handled separately, preserving columnar storage benefits without memory constraints.
3Quantity of substance
If the size of column values is limited to 16 MB, 32 MB, or 64 MB, then memory bloat is prevented, but user storage needs are not met
Solution Approach 1:
The patent introduces auxiliary storage as an intermediary between the row group storage and the user's large column values. This intermediary layer allows the database to store arbitrarily large column values while maintaining efficient memory management for row groups, satisfying both memory constraints and user storage needs.
Data Source
AI summary
Efficient large column values storage in columnar databases is disclosed, including: providing a database comprising a write storage, a read storage, and an auxiliary storage, and wherein the auxiliary storage is used to store a large column value of a row that is associated with either the write storage or the read storage, wherein the large column value comprises a column value that meets a set of large column value criteria; and operating the database using the write storage, the read storage, and the auxiliary storage.