Metadata-Free ETL Job Execution via Message Passing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
ETL tools require significant computing resources to maintain and retrieve metadata from disk storage, which hinders their efficiency and scalability, especially in resource-constrained environments like cloud-based systems.
Innovation Solution
An ETL system where a coordinator provides job information as a message to the ETL engine, eliminating the need for direct access to a disk repository, thereby reducing resource consumption and enabling lightweight operation across various devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ETL tools use a metadata repository stored in disk database, then they can maintain and retrieve job information, but they require significant computing resources and time to maintain and retrieve metadata
Solution Approach 1:
The patent extracts the metadata repository requirement from the ETL tool by using a message-passing mechanism. Instead of requiring the ETL tool to directly access and manage a disk-based metadata repository, the system passes job information as messages between components, eliminating the need for local metadata storage and retrieval operations.
Solution Approach 2:
The patent introduces a message-passing intermediary mechanism that mediates between the coordination component and the ETL tool. The job information is transmitted through messages rather than direct database access, serving as an intermediary that eliminates the need for the ETL tool to maintain or access disk-based metadata repositories.
2Ease of operation
If ETL tools access metadata from disk storage, then they can retrieve job information, but additional resources are required to read and collect metadata before executing jobs
Solution Approach 1:
The patent extracts the disk I/O operation from the ETL tool's workflow by implementing a message-passing system. Job information is passed through memory-based message queues rather than being read from disk storage, eliminating the energy-consuming disk read and metadata collection operations.
Solution Approach 2:
The patent prepares job information in advance as structured messages that can be directly consumed by the ETL tool. By pre-formatting job information as passable messages rather than storing it in disk databases, the system eliminates the need for runtime metadata reading and collection operations.
3Adaptability or versatility
If ETL tools are designed to work with metadata repositories, then they can perform data extraction and transformation, but they cannot be easily deployed on devices with limited computing resources
Solution Approach 1:
The patent extracts the heavy metadata repository infrastructure from the ETL tool deployment requirements. By using a message-passing mechanism, the ETL tool no longer requires local disk-based metadata storage, enabling deployment on devices with limited computing resources and simplifying the system setup.
Solution Approach 2:
The patent enables the ETL tool to receive job information through messages without requiring it to maintain or access external metadata repositories. This self-service approach where job information is passed directly as messages reduces the ETL tool's infrastructure requirements and improves deployability across diverse devices.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for a an ETL (extract-transform-load) system. An embodiment operates by receiving, at a processor, a message including a request to move data from a source database to a target database. The data is retrieved from the source database. One or more operations to perform on the data that convert the data from a source format associated with the source database to a target format associated with the target database are determined from the message. The one or more operations are executed on the data. The data is stored on the target database in the target format.