Disk-Based Columnar Storage for Text Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face challenges in efficiently supporting text analysis of tables stored in extended disk-centric columnar storage, as tables with full-text indexes cannot be stored in extended storage, limiting the integration of text analysis with disk-based columnar storage.
Innovation Solution
The system allows database tables associated with full-text indexes to be stored in disk-based columnar storage, with text analysis results also stored within this storage, enabling flexible partitioning between in-memory and disk-based storage to address performance, cost, and capacity concerns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tables with full-text indexes are stored in in-memory columnar storage, then text analysis can be performed, but storage cost increases and capacity is limited
Solution Approach 1:
The patent segments the storage system into two distinct parts: in-memory columnar storage for active text analysis operations and extended disk-centric columnar storage for archival and cold data. This segmentation allows tables requiring text analysis to remain in memory while other tables move to disk-based extended storage, resolving the contradiction between maintaining text analysis capability and increasing storage capacity.
Solution Approach 2:
The patent introduces an intermediary mechanism - the full-text index structure that can reference data in extended storage without requiring the entire table to reside in memory. The full-text index acts as a mediator, enabling text analysis operations on tables stored in extended disk-centric columnar storage by maintaining only the necessary index structures in memory.
2Quantity of substance
If tables are moved to extended disk-centric columnar storage, then storage capacity and cost efficiency improve, but text analysis capability is lost
Solution Approach 1:
The patent applies preliminary action by pre-building and maintaining full-text indexes on tables stored in extended storage. These indexes are constructed in advance and stored in a format that enables efficient text analysis operations. When text analysis is needed, the pre-computed indexes are loaded into memory, allowing the actual analysis to proceed without requiring the entire table to be in memory.
Solution Approach 2:
The patent uses copying by creating full-text index structures that replicate the essential text searchability information from the original tables stored in extended storage. Rather than copying entire tables into memory, only the compressed and indexed text representations are copied, enabling text analysis while preserving the space-efficient extended storage arrangement.
3Speed
If all tables are kept in in-memory storage, then text analysis performance is high, but storage cost increases
Solution Approach 1:
The patent applies local quality by assigning different storage characteristics to different tables or table partitions based on their access patterns and text analysis requirements. Frequently accessed tables requiring text analysis maintain full in-memory presence with complete indexes, while infrequently accessed tables use extended disk-centric storage with optimized index structures, optimizing both performance and cost locally for each table.
Solution Approach 2:
The patent introduces dynamics by enabling flexible movement of tables between in-memory and extended storage environments based on changing requirements. Tables can be dynamically promoted from extended storage to in-memory storage when text analysis performance becomes critical, and demoted back when requirements change, allowing the system to adapt to varying workloads and optimize the performance-cost tradeoff over time.
Data Source
AI summary
A database system includes a data server, a disk-based storage system in communication with the data server, a source table including a plurality of columns, where data of the source table is stored in the disk-based storage system in columnar format, a full-text index associated with one of the plurality of columns, and a text analysis result table associated with the one of the plurality of columns, where the data of the text analysis result table is stored in the disk-based storage system in columnar format.