In-Database Data Provisioning and Metadata Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analysis methods for big data stored in databases are inefficient due to the need for data transfer between the database and application layers, which hinders real-time analysis and search operations, and lack effective utilization of database parallelization capabilities.
Innovation Solution
In-database processing techniques are adopted, where data and metadata tables are exposed for searching and analysis, allowing processing logic to reside within the database layer, leveraging its parallelization capabilities and reducing unnecessary data movement, with metadata evaluation procedures determining column classifications and hierarchies to optimize data analysis and search operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transferred between database and application layers for analysis, then data can be accessed by external tools, but processing time and data movement overhead increase
Solution Approach 1:
The patent merges the data storage function (database layer) with the data analysis function (application layer) by exposing metadata tables directly within the database. This allows search tools to perform analysis operations directly on the stored data without external data transfer, combining both functions in one location to eliminate time loss while maintaining accessibility.
2Adaptability or versatility
If a thick middle layer is introduced for data processing and replication, then data can be transformed and indexed, but system complexity and processing overhead increase
Solution Approach 1:
The patent extracts the essential data processing capability by exposing metadata tables directly within the database layer, removing the need for a thick middle layer. The metadata tables contain classification information, hierarchy structures, and indexing data that enable search tools to perform transformations and analyses directly, eliminating the complex intermediary processing layer while retaining adaptability.
3Speed
If data is replicated into search indexes externally, then search operations can be performed, but data transfer and index update overhead increase
Solution Approach 1:
The patent implements self-service by having the database layer automatically maintain and expose metadata tables that contain pre-processed classification and hierarchy information. Search tools can directly query these metadata tables within the database without external replication, allowing the system to serve its own indexing needs internally and eliminate energy-consuming data transfer operations.
4Productivity
If manual identification of data structures is performed, then data analysis can be accomplished, but analysis time and manual effort increase
Solution Approach 1:
The patent applies preliminary action by pre-processing data during the storage phase, where metadata tables are automatically generated and populated with classification information, hierarchy structures, and indexing data. This preliminary organization of data structures eliminates the need for manual identification during analysis, enabling search tools to quickly query pre-organized metadata and significantly reducing analysis time while maintaining high productivity.
Data Source
AI summary
A user uploads date sets through a client to a database. The data sets are provisioned in the database for in-database searching. The data sets are evaluated and classifications for the columns of the tables that include the data set are detected. Columns content may be classified into different analysis types, aggregation types, formats, categories, hierarchies, etc. Metadata is generated based on the evaluation of the data sets. A schema is used to store the metadata that describes the detected classification of the columns. The schema is stored in the database and is used when a search in the database is performed.