In-Memory Data Profiling via Integrated Database Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data profiling is a slow process due to the time-consuming transfer of data from sources to memory, especially as data volume grows, limiting the efficiency of table-wide or cross-table profiling.
Innovation Solution
Implementing data profiling in-memory within the database management system, utilizing an in-memory database that stores data primarily in main memory, allowing for parallel processing across multiple clusters and reducing the need for data transfer, with components like index servers and statistics servers performing data attribute calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is transferred from source to memory for profiling, then data profiling can be performed, but the process becomes slow especially as data volume grows
Solution Approach 1:
The patent merges the data storage function and data profiling function into a single integrated system. The in-memory database system stores data in main memory and performs profiling operations directly on the stored data, eliminating the need to transfer data between separate storage and processing systems. This consolidation resolves the technical contradiction by maintaining full data profiling capability while eliminating time-consuming data transfer operations.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of an in-memory database buffer that resides within the database system itself. This buffer serves as an intermediate storage location that allows profiling operations to be performed without extracting data to external systems. The intermediary in-memory buffer resolves the contradiction by enabling profiling access to data while avoiding the time penalty of traditional data extraction and transfer processes.
2Quantity of substance
If data profiling is performed on large datasets, then comprehensive profiling results are achieved, but processing speed decreases
Solution Approach 1:
The patent changes the physical state parameter of data storage from traditional disk-based storage to in-memory storage. By altering the storage medium parameter to main memory, the system achieves both the ability to handle large data volumes and maintain high processing speeds. The in-memory database buffer allows comprehensive profiling of large datasets while avoiding the speed degradation that would occur with traditional storage and transfer mechanisms.
Data Source
AI summary
In an example embodiment, a method of performing data profiling in an in-memory database is provided. A command requesting a data attribute calculation on data in the in-memory database may be received. A precreated procedure stored in the in-memory database may then be used to perform the data attribute calculation on the data in the in-memory database The precreated procedure may be described in a computer language utilized by the in-memory database. Finally, a result of the performing of the data attribute calculation may be returned to the client computer.