Regular Expression Indexing for Columnar Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Column-oriented data storage formats, such as Apache Parquet, require scanning entire files or columns to locate specific data, leading to high CPU, IO, and memory usage, especially when dealing with large files and sensitive information retrieval needs like GDPR compliance.
Innovation Solution
Implementing regular expression indexes within metadata for each row group in columnar data storage formats, allowing for efficient identification and retrieval of specific data by directly accessing and evaluating only relevant column pages and row groups that match the query criteria, thus conserving computing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If entire files or columns are scanned to locate specific data in column-oriented storage format, then data can be found, but CPU time, IO, and memory usage increase significantly
Solution Approach 1:
The patent applies preliminary action by creating and storing regular expression indexes in advance during the data writing phase. These indexes are built column-by-column and stored in the file metadata, so when a query arrives, the system can directly use the pre-computed indexes to locate matching data without scanning entire columns, thus resolving the contradiction between accurate data location and query efficiency
Solution Approach 2:
The patent introduces regular expression indexes as an intermediary layer between the stored data and query operations. These indexes act as a mediator that captures the essential patterns of data without requiring full data scanning, enabling efficient query processing while maintaining accurate data location capabilities
2Productivity
If column-oriented data storage format is used to organize data into columns, then query performance is optimized and I/O requirements are reduced, but the size of each column becomes significantly large requiring full column scans
Solution Approach 1:
The patent extracts the essential searchability information from the large column data by creating compact regular expression indexes. These indexes are stored separately in metadata and contain only the necessary pattern-matching information, allowing the system to work with a small extracted representation rather than the full large-volume column data during queries
3Loss of time
If personal information needs to be located and deleted quickly to adhere to data protection standards, then compliance is achieved, but scanning entire files consumes large amount of CPU time, IO, and memory
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing regular expression indexes that capture personal information patterns during data ingestion. When deletion or retrieval operations are needed for compliance, the system uses these pre-built indexes to quickly identify relevant data without scanning entire files, thus reducing both time and computing resource consumption while achieving regulatory compliance
Data Source
AI summary
Techniques are described herein for indexing personal information in columnar data storage format based files. In an embodiment, row groups of rows that comprise a plurality of columns are stored in a set of files. Each column of a row group is stored in a chunk of column pages in the set of files. A regular expression index that indexes a particular column in the set of files is stored for each row group. The regular expression index identifies column pages in the chunk of the particular column that include a particular column value that satisfies a regular expression specified in a query. The regular expression specified in the query in evaluated against the particular column using the regular expression index.