Integrated Metadata Store for Faster Data Warehouse Building
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems lack efficient methods for generating and integrating metadata during data transformation processes, leading to inefficiencies in building data warehouses.
Innovation Solution
A data processing system that utilizes templates for creating dimension and fact tables, backed by class infrastructure, which includes metadata generation and storage capabilities, reducing the time and complexity of building data warehouses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional unstructured processes are used for building data warehouses, then flexibility in customization is maintained, but the time required to build data warehouses increases significantly
Solution Approach 1:
The patent segments metadata into distinct categories including process definition metadata, runtime metadata, and integration metadata. Each metadata type is stored in separate structured formats within the data warehouse, allowing efficient retrieval and management while reducing the time required to build and query data warehouses.
Solution Approach 2:
The system performs preliminary actions by automatically generating and storing process definition metadata and runtime metadata during the data processing workflow execution. This pre-structured metadata is prepared in advance and organized in the data warehouse before queries are executed, eliminating the need for time-consuming manual metadata creation and structuring during data warehouse construction.
2Productivity
If integrated metadata generation and storage is implemented, then data management efficiency is improved, but the complexity of the data processing system increases
Solution Approach 1:
The patent merges metadata generation and storage functions directly into the data processing system workflow. The metadata generation component is integrated with the data processing components, allowing runtime metadata to be generated and stored automatically during normal data processing operations without requiring separate external systems, thereby improving efficiency while managing complexity through integration.
Solution Approach 2:
The system implements self-service by automatically generating process definition metadata and runtime metadata without requiring manual intervention. The data processing system itself produces and manages its metadata through integrated components that automatically capture processing information and store it in the data warehouse, eliminating the need for external metadata management tools or manual metadata creation processes.
3Productivity
If structured templates are used for creating dimension and fact tables, then the time required to build data warehouses is reduced, but the ease of operation decreases due to stricter formatting requirements
Solution Approach 1:
The system applies self-service by using automated query generation components that read the structured metadata from the data warehouse and automatically generate appropriate SQL queries for creating dimension and fact tables. This automation eliminates the need for manual query writing while maintaining strict structural requirements, thereby reducing the time required to create tables without significantly impacting ease of operation since the process is automated.
Data Source
AI summary
A method of generating and storing metadata in a data processing system includes defining metadata sets of process definition metadata based on a process definition of the data processing system. The metadata sets include a first set of metadata corresponding to a processing step of the data processing system, a second set of metadata corresponding to a processing step successor, and a third set of metadata corresponding to a data object that is produced or consumed by the processing step in the data processing system. The method further includes executing one or more steps of the data processing system according to the process definition and generating runtime metadata during an execution of the one or more data processing system steps. The method further includes storing the runtime metadata and forming a metadata data store that integrates the runtime metadata and the metadata sets of process definition metadata.