Database Table Categorization for Selective Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data integration projects, accessing and processing large datasets from source databases with thousands of tables is challenging due to the lack of documented data models and limited time windows, where terabytes of data need to be processed efficiently.
Innovation Solution
A method is provided to categorize data tables in a source database using predefined metrics and characteristics, allowing for selective data processing by identifying table categories such as master, transactional, reference, metadata, history/audit, and queue tables, thereby prioritizing relevant data for processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data tables are processed to ensure complete data extraction, then data completeness is improved, but processing time and computational resources are excessively consumed
Solution Approach 1:
The patent segments the large set of data tables into distinct categories (master tables, transactional tables, reference tables, metadata tables, history/audit tables, queue tables) based on their functional characteristics. This segmentation allows the system to apply different processing strategies to different categories, processing only relevant categories based on the specific data integration project requirements, thereby maintaining data completeness where needed while significantly improving processing speed by excluding irrelevant tables.
2Measurement precision
If comprehensive data processing is performed on all tables, then data accuracy is improved, but the amount of data to be processed increases exponentially
Solution Approach 1:
The patent extracts and identifies only the relevant subset of data tables needed for the specific data integration project by evaluating tables against category-specific metrics and characteristics. This extraction process removes unnecessary tables from the processing scope, thereby maintaining data accuracy for relevant tables while dramatically reducing the total data volume that requires processing, storage, and transformation.
3Adaptability or versatility
If all table types are included in data processing, then completeness of data model coverage is improved, but hardware requirements and computational overhead increase significantly
Solution Approach 1:
The patent implements a dynamic table selection mechanism that adapts the scope of data processing based on the specific requirements of each data integration project. By using configurable metrics and characteristics for each table category, the system can dynamically adjust which tables are processed, ensuring comprehensive coverage of relevant data models while avoiding the fixed overhead of processing all possible table types, thereby reducing hardware requirements and computational complexity.
Data Source
AI summary
Embodiments relate to accessing a set of data tables in a source database. A set of table categories is provided for tables in the source database and a set of metrics is provided. For each table of the set of the data tables: the set of metrics is evaluated, the evaluated set of metrics is analyzed, and the table is categorized into one of the set of table categories using the result of the analysis. Information indicative of the table category of each table of the set of tables is output, and in response, a request to select data tables of the set of data tables is received according to a part of the table categories for data processing. A subset of data tables of the set of data tables is selected using the table categories for performing the data processing on the subset of data tables.