Columnar Database Featurization via Augmented Dictionary
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional approaches for accessing columnar databases with artificial intelligence (AI) algorithms require a multi-step process involving relational database queries, data encoding, and featurization, leading to inefficiencies and increased processing resources and latency.
Innovation Solution
Implementing columnar databases with augmented dictionary values and count information to directly provide featurized data, allowing for direct data access and reducing the need for post-retrieval transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional multi-step process (relational database queries, data encoding, featurization) is used to access columnar databases with AI algorithms, then data can be accessed and transformed, but processing resources increase and latency increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing featurized data directly in the columnar database during data ingestion. Instead of performing featurization transformations at query time, the system performs these transformations in advance and stores the results, eliminating the need for repeated processing and reducing both computational overhead and latency.
Solution Approach 2:
The patent merges multiple separate processing steps (data encoding, featurization, and data retrieval) into a single integrated operation. By combining these steps and pre-computing features during data loading, the system eliminates the need for sequential multi-step processing when AI algorithms need data, thereby improving productivity and reducing latency.
2Productivity
If traditional multi-step process is used, then data transformations can be performed, but processing resources increase
Solution Approach 1:
The system performs data encoding and featurization transformations in advance during data ingestion, storing the pre-processed data in columnar format. This eliminates the need to repeat resource-intensive transformations every time data is accessed by AI algorithms, significantly reducing processing resource consumption while maintaining high data access efficiency.
Solution Approach 2:
The patent extracts and separates the computationally intensive featurization step from the data retrieval operation. By performing featurization in advance and storing only the results in the columnar database, the system removes the burden of repeated transformations during AI data access, thereby improving productivity without increasing processing resources.
3Adaptability or versatility
If post-retrieval transformations are performed, then data can be adapted for AI algorithms, but additional processing steps are required
Solution Approach 1:
The system performs data adaptability transformations in advance during data loading, converting data into AI-ready formats and storing it in columnar structure. This eliminates the need for post-retrieval transformations and reduces the number of processing steps required when AI algorithms access the data, while maintaining full adaptability to different AI framework requirements.
Data Source
AI summary
A database service that receives and manages data into columnar databases. Individual columnar databases or columnar database partitions are associated with a dictionary that provides an encoding for the data in the columnar database. The database service generates count information for the dictionary entries. The database service also maintains one or more augmented dictionary values that correspond to featurization criteria. The database service applies the augmented dictionary values to respond to requests for feature data.